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Figure 7.6: Zerg vs Protoss time of the third Hatch and first appearance of Mutalisks. The 
bottom left cluster (square data points) is the one labeled as mutas. Variable volume, variable 
shape, variable orientation covariance matrices. 
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7.5 Build tree prediction 



The work described in the next two sections can be classified as probabilistic plan recognition. 
Strictly speaking, we present model-based machine learning used for prediction of plans, while 
our model is not limited to prediction. We performed two levels of plan recognition, both are 
learned from the replays: tech tree prediction (unsupervised, it does not need openings labeling, 
this section) and opening prediction (semi-supervised or supervised depending on the labeling 
method, next section). 



7.5.1 Bayesian model 
Variables 

• BuildTree: BT £ {$ , {buildingi} , {building2} , {buildingi A building2}, ■ ■ ■ }■ all the pos- 
sible building trees for the given race. For instance {pylon, gate} and {pylon, gate, core} 
are two different BuildTrees. 



• Observations: Ojg[i...j\r| € {0, 1}, Ok is 1/true if we have seen (observed) the feth building 
(it can have been destroyed, it will stay "seen"). Otherwise, it is 0/ false. 

• A € {0, 1}: coherence variable (restraining BuildTree to possible values with regard to 
O v .n) 

• Time: T G [1 . . . PJ, time in the game (1 second resolution). 



At first, we generated all the possible (according to the game rules) build trees" (BT val- 
ues) of StarCraft, and there are between ~ 500 and 1600 depending on the race without even 
counting buildings duplicates! We observed that a lot of possible build trees are too absurd to 
be performed in a competitive match and were never seen during the learning. So, we restricted 
BT to have its value in all the build trees encountered in our replays dataset and we added 
multiple instances of the basic unit producing buildings (gateway, barracks), expansions and 
supply buildings (depot, pylon, "overlord" as a building), as shown in Table 7.2 This way, there 
are 810 build trees for Terran, 346 for Protoss and 261 for Zerg (learned from ~ 3000 games for 
each race, see Table 7.1). In a new game, if we encounter a build tree that we never saw, we 
are in a unknown state. Anyway we would not have seen enough data (any at all) during the 
learning to conclude anything. We could look at the proximity of the build tree to other known 
build trees, see section 7.5.2 and the Discussion (7.5.3). 



Decomposition 

The joint distribution of our model is the following: 

N 



P(T, BT, Oi . . . O n , A) = P(T\BT)P(BT)P(X\BT, 0 1:N ) J] P(0;) (7.1) 



i=l 



This can also be see as a plate diagram in Figure 7.7 
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Figure 7.7: Graphical model (plate notation) of the build tree Bayesian model, gray variables 
are known. 

Forms 

• P(BT) is the prior distribution on the build trees. As we do not want to include any bias, 
we set it to the uniform distribution. 

• P(Oi : Ar) is unspecified, we put the uniform distribution (we could use a prior over the most 
frequent observations). 

• P(X\BT, 0\-n) is a functional Dirac that restricts BT values to the ones than can co-exist 
with the observations: 

P(A = l\bt, oi-n) = 1 if bt can exist with oi-n 

< 

P(A = l\bt,o 1:N ) = 0 else 

A build tree value (bt) is compatible with the observations if it covers them fully. For 
instance, BT = {pylon, gate, core} is compatible with o core = 1 but it is not compatible 
with Of or g e = 1. In other words, buildTree is incompatible with o\-,n iff {o\-n\{oi-n A 
buildTree}} ^ 0. 

• P(T\BT = bt) = M{ji bt ,crl t ): P(T\BT) are discrete normal distributions ("bell shapes"). 
There is one bell shape over T per bt. The parameters of these discrete Gaussian distri- 
butions are learned from the replays. 



Identification (learning) 

As we have full observations in the training phase, learning is just counting and averaging. The 
learning of the P(T\BT) bell shapes parameters takes into account the uncertainty of the bt for 
which we have few observations: the normal distribution P(T\bt) begins with a high a^ t , and 
not a strong peak at on the seen T value and sigma = 0. This accounts for the fact that 
the first(s) observation(s) may be outlier (s). This learning process is independent on the order 
of the stream of examples, seeing point A and then B or B and then A in the learning phase 
produces the same result. 
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Questions 

The question that we will ask in all the benchmarks is: 



N 



P(BT\T = t, 0 1:N = 01:7V, A = 1) oc P(t\BT)P(BT)P(X\BT, o 1:N ) [] P(o t ) (7.2) 



i=l 



An example of the evolution of this question with new observations is depicted in Figure |7.8[ in 
which we can see that build trees (possibly closely related) succeed each others (normal) until 
convergence. 



Bayesian program 

The full Bayesian program is: 



Variables 
T,BT,O l ...O N ,\ 
Decomposition 
P(T, BT, 0 1 ...0 N ,\) = JD 
= P(X\BT, O lu _ m )P(T\BT) illi P(O t )P(BT) 
Forms 

P(X\BT, Ofi...iv]) = functional Dirac (coherence) 
P(T\BT = bt) = discrete Af(u bt , of t ) 
Identification 

p(gr=mop t = q P ) = iSSSSg) 

S^buVbt) = argmax Mi(T P(T|Br = bt;^a 2 ) 
Question 

P(BT\T = t,0 1:N = o 1:N ,X= 1) cx P(t\BT)P(BT)P(X\BT,o 1:N )llti'P(o i 



ofi 
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Pi \ 
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7.5.2 Results 

All the results presented in this section represents the nine match-ups (races combinations) in 1 
versus 1 (duel) of StarCraft. We worked with a dataset of 8806 replays (~ 1000 per match-up) 



of skilled human players (see 7.1) and we performed cross-validation with 9/10th of the dataset 
used for learning and the remaining l/10th of the dataset used for evaluation. Performance wise, 
the learning part (with ~ 1000 replays) takes around 0.1 second on a 2.8 Ghz Core 2 Duo CPU 
(and it is serializable) . Each inference (question) step takes around 0.01 second. The memory 
footprint is around 3Mb on a 64 bits machine. 

Metrics 

k is the numbers of buildings ahead that we can accurately predict the build tree of the opponent 
at a fixed d. d values are distances between the predicted build tree(s) and the reality (at fixed 
k). 
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Observations (#buildings seen) 

Figure 7.8: Evolution of P(BT\observations,time, A = 1) in time (seen/observed buildings 
on the x-axis). Only BT with a probability > 0.01 are shown. The numbers in the legend 
correspond to build trees identifier numbers. The interpolation was obtained by fitting a second 
order polynomial. 

The robustness to noise is measured by the distance d to the real build tree with increasing 
levels of noise, for k = 0. 

The predictive power of our model is measured by the k > 0 next buildings for which we 
have "good enough" prediction of future build trees in: 

P(BT t+k \T = t, 0\-,N = Ol : 7V, A = 1) 

"Good enough" is measured by a distance d to the actual build tree of the opponent that 
we tolerate. We used a set distance: d(bt\,bt2) = card(&iiA6i2) = card((6ii \J bt2)\(bti f] bt%)). 
One less or more building in the prediction is at a distance of 1 from the actual build tree. The 
same set of buildings except for one replacement is at a distance of 2 (that would be 1 is we 
used tree edit distance with substitution). 

• We call d(best, real) — 'best" the distance between the most probable build tree and the 
one that actually happened. 

• We call d(bt,real) * P(6t)— 'mean" the marginalized distance between what was inferred 
balanced (variable bt) by the probability of inferences (P (&£)). 

Note that this distance is always over the complete build tree, and not only the newly inferred 
buildings. This distance metric was counted only after the fourth (4th) building so that the 
first buildings would not penalize the prediction metric (the first building can not be predicted 
4 buildings in advance). 

For information, the first 4 buildings for a Terran player are more often amongst his first 
supply depot, barracks, refinery (gas), and factory or expansion or second barracks. For Zerg, 
the first 4 buildings are is first overlord, zergling pool, extractor (gas), and expansion or lair tech. 
For Protoss, it can be first pylon, gateway, assimilator (gas), cybernectics core, and sometimes 
robotics center, or forge or expansion. 
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Table 7.4 shows the full results, the first line has to be interpreted as "without noise, the 
average of the following measures are" (columns): 



d for k = 0: 

— d(best,real) = 0.535, it means that the average distance from the most probable 
("best") build tree to the real one is 0.535 building. 

— d(bt,real) * P(bt) = 0.870, it means that the average distance from each value bt of 
the distribution on BT, weighted by its own probability (P(6f)), to the real one is 
0.87 building. 

k for d = 1: 

— best k = 1.193, it means that the average number of buildings ahead (of their con- 
struction) that the model predicts at a fixed maximum error of 1 (d = 1) for the most 
probable build tree ("best") is 1.193 buildings. 

— "mean" k = 3.991, it means that the average number of buildings ahead (of their 
construction) that the model predicts at a fixed maximum error 1 (d = 1), summed 
for bt in BT value and weighted by P(bt) is 3.9910 

as for the bullet above but k for d = 2 and d = 3. 



The second line ("min") is the minimum across the different match-upsj " (as they are detailed for 



noise = 10%), the third ("max") is for the maximum across match-ups, both still at zero noise. 
Subsequent sets of lines are for increasing values of noise (i.e. missing observations). 



Predictive power 

To test the predictive power of our model, we are interested at looking at how big k is (how 
much "time" before we can predict a build tree) at fixed values of d. We used d = 1, 2, 3: with 
d = 1 we have a very strong sense of what the opponent is doing or will be doing, with d = 3, 
we may miss one key building or the opponent may have switched of tech path. 

We can see in Table 7.4 that with d = 1 and without noise (first line), our model predicts 
in average more than one building in advance (k > 1) what the opponent will build next if we 
use only its best prediction. If we marginalize over BT (sum on BT weighted by P(bt)), we 
can almost predict four buildings in advance. Of course, if we accept more error, the predictive 
power (number of buildings ahead that our model is capable to predict) increases, up to 6.122 
(in average) for d = 3 without noise. 



Robustness to missing informations ("noise") 

The robustness of our algorithm is measured by the quality of the predictions of the build trees 
for k = 0 (reconstruction, estimation) or k > 0 (prediction) with missing observations in: 

P(BT t+k \T = t, 0 1:N = partial(o 1:N ), A = 1) 

6 This shows that when the most probable build tree is mistaken, there is 1) not much confidence in it 
(otherwise we would have P(btbe S t) ~ 1-0 and the "mean" k for fixed d values would equal the "best" one). 2) 
much information in the distribution on BT, in the subsequent P(bt^t, es t) values. 
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Table 7.4: Summarization of the main results/metrics, one full results set for 10% noise 





measure 




d for k = 


= 0 


k for d — 1 


k for d = 2 


k for d = 3 


noise 




d(best, real) d(bt, 


real) * P(bt) 


best 


"mean" 


best 


"mean" 


best 


"mean" 


if? 

O 


average 


0.535 




0.870 


1.193 


3.991 


2.760 


5.249 


3.642 


6.122 


min 


0.313 




0.574 


0.861 


2.8 


2.239 


3.97 


3.13 


4.88 




max 


1.051 




1.296 


2.176 


5.334 


3.681 


6.683 


4.496 


7.334 


£S 


PvP 


0.397 




0.646 


1.061 


2.795 


2.204 


3.877 


2.897 


4.693 


o 


PvT 


0.341 




0.654 


0.991 


2.911 


2.017 


4.053 


2.929 


5.079 




PvZ 


0.516 




0.910 


0.882 


3.361 


2.276 


4.489 


3.053 


5.308 




TvP 


0.608 




0.978 


0.797 


4.202 


2.212 


5.171 


3.060 


5.959 




TvT 


1.043 




1.310 


0.983 


4.75 


3.45 


5.85 


3.833 


6.45 




TVZ 


0.890 




1.250 


1.882 


4.815 


3.327 


5.873 


4.134 


6.546 




ZvP 


0.521 




0.933 


0.89 


3.82 


2.48 


4.93 


3.16 


5.54 




ZvT 


0.486 




0.834 


0.765 


3.156 


2.260 


4.373 


3.139 


5.173 




ZvZ 


0.399 




0.694 


0.9 


2.52 


2.12 


3.53 


2.71 


4.38 




average 


0.578 




0.912 


1.017 


3.592 


2.483 


4.683 


3.213 


5.459 




min 


0.341 




0.646 


0.765 


2.52 


2.017 


3.53 


2.71 


4.38 




max 


1.043 




1.310 


1.882 


4.815 


3.45 


5.873 


4.134 


6.546 


£8 


average 


0.610 




0.949 


0.900 


3.263 


2.256 


4.213 


2.866 


4.873 


O 

CN 


min 


0.381 




0.683 


0.686 


2.3 


1.858 


3.25 


2.44 


3.91 




max 


1.062 




1.330 


1.697 


4.394 


3.133 


5.336 


3.697 


5.899 


g? 


average 


0.670 




1.003 


0.747 


2.902 


2.055 


3.801 


2.534 


4.375 


o 

CO 


min 


0.431 




0.749 


0.555 


2.03 


1.7 


3 


2.22 


3.58 




max 


1.131 




1.392 


1.394 


3.933 


2.638 


4.722 


3.176 


5.268 


£8 


aerage 


0.740 




1.068 


0.611 


2.529 


1.883 


3.357 


2.20 


3.827 


o 


min 


0.488 




0.820 


0.44 


1.65 


1.535 


2.61 


1.94 


3.09 




max 


1.257 




1.497 


1.201 


3.5 


2.516 


4.226 


2.773 


4.672 




average 


0.816 




1.145 


0.493 


2.078 


1.696 


2.860 


1.972 


3.242 


o 


min 


0.534 




0.864 


0.363 


1.33 


1.444 


2.24 


1.653 


2.61 




max 


1.354 




1.581 


1 


2.890 


2.4 


3.613 


2.516 


3.941 


g? 


average 


0.925 




1.232 


0.400 


1.738 


1.531 


2.449 


1.724 


2.732 


o 

to 


min 


0.586 




0.918 


0.22 


1.08 


1.262 


1.98 


1.448 


2.22 




max 


1.414 




1.707 


0.840 


2.483 


2 


3.100 


2.083 


3.327 


g? 


average 


1.038 




1.314 


0.277 


1.291 


1.342 


2.039 


1.470 


2.270 


o 
t- 


min 


0.633 




0.994 


0.16 


0.79 


1.101 


1.653 


1.244 


1.83 




max 


1.683 




1.871 


0.537 


1.85 


1.7 


2.512 


1.85 


2.714 


££ 


average 


1.134 




1.367 


0.156 


0.890 


1.144 


1.689 


1.283 


1.831 


o 


min 


0.665 




1.027 


0.06 


0.56 


0.929 


1.408 


1.106 


1.66 




max 


1.876 




1.999 


0.333 


1.216 


1.4 


2.033 


1.5 


2.176 



The "reconstructive" power (infer what has not been seen) ensues from the learning of our 
parameters from real data: even in the set of build trees that are possible, with regard to the 
game rules, only a few will be probable at a given time and/or with some key structures. The 
fact that we have a distribution on BT allows us to compare different values bt of BT on the 
same scale and to use V{BT) ("soft evidence") as an input in other models. This "reconstructive" 
power of our model is shown in Table 7.4 with d (distance to actual building tree) for increasing 
noise at fixed k = 0. 



Figure 7.9 displays first (on top) the evolution of the error rate in reconstruction (distance 
to actual building) with increasing random noise (from 0% to 80%, no missing observations to 8 
missing observations over 10). We consider that having an average distance to the actual build 
tree a little over 1 for 80% missing observations is a success. This means that our reconstruction 
of the enemy build tree with a few rightly timed observations is very accurate. We should ponder 
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that this average "missed" (unpredicted or wrongly predicted) building can be very important 
(for instance if it unlocks game changing technology). We think that this robustness to noise 
is due to P(T\BT) being precise with the amount of data that we used, and the build tree 
structure. 



Secondly Figure 7.9 displays (at the bottom) the evolution of the predictive power (number 
of buildings ahead from the build tree that it can predict) with the same increase of noise. 
Predicting 2 building ahead with d = 1 (1 building of tolerance) gives that we predict right (for 
sure) at least one building, at that is realized up to almost 50% of noise. In this case, this "one 
building ahead" right prediction (with only 50% of the information) can give us enough time to 
adapt our strategy (against a game changing technology). 



7.5.3 Possible improvements 

We recall that we used the prediction of build trees (or tech trees) , as a proxy for the estimation 
of an opponent's technologies and production capabilities. 

This work can be extended by having a model for the two players (the bot/AI and the 
opponent): 

P(BTbot, BT op , O oPi i : 7v, T, A) 
So that we could ask this (new) model: 

P(BT bot \obs op ,i; N ,t,\ = 1) 

This would allow for simple and dynamic build tree adaptation to the opponent strategy (dy- 
namic re-planning), by the inference path: 

P(BT bat \obs oP)1:N , t,X = l) 
oc ^2 P(BT bot \BT op ) (learned) 

BT op 

xP(BT op )P(o opA ; N ) (priors) 
xP(X\BT op , o op .i;7v) (consistency) 
xP(t\BT op ) (learned) 

That way, one can ask "what build/tech tree should I go for against what I see from my op- 
ponent", which tacitly seeks the distribution on BT op to reduce the complexity of the possible 
combinations of 0\-n- It is possible to not marginalize over BT op , but consider only the most 
probable(s) BT op . In this usage, a filter on BT bo t (as simple as P(BT b t ot \BT^~ t 1 ) can and should 
be added to prevent switching build orders or strategies too often. 



137 



— 18 

en 
C 

~ 16 



1.4 



0) 



distance to real tech tree with increasing noise 



• • error/distance best 

- - interp. error best 

• * error/distance marginal 

- - interp. error marginal 



1.0 



0.8 



o 

(U 

u 
c 

03 

2 0.6 

jo 

O 0.4 

k_ 
1_ 
LU 



0,2 




0% 10% 50% 

Noise ratio (missing observations) 



80% 



marginal ("mean") predictive power with increasing noise 



0) 

o 

Q_ 

> 



• ♦ k with d=l 

- - interpolation 

• • k with d=2 

- - interpolation 

• ♦ k with d=3 

- - interpolation 



-v 



0% 10% 50% 

Noise ratio (missing observations) 



80% 



Figure 7.9: Evolution of our metrics with increasing noise, from 0 to 80%. The top graphic 
shows the increase in distance between the predicted build tree, both most probable ("best") 
and marginal ("mean") and the actual one. The bottom graphic shows the decrease in predictive 
power: numbers of buildings ahead (k) for which our model predict a build tree closer than a 
fixed distance/error (d). 
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7.6 Openings 
7.6.1 Bayesian model 



We now build upon the previous build tree" predictor model to predict the opponent's strategies 



(openings) from partial observations. 

Variables 

As before, we have: 

• Build trees: BT G [®,buildingi,building2,buildingi A building2,techtrees, . . .]: all 
the possible building trees for the given race. For instance {pylon, gate} and 
{pylon, gate, core} are two different BT. 

• N Observations: Ojgp _m G {0, 1}, is 1 (true) if we have seen (observed) the kth. 
building (it can have been destroyed, it will stay "seen"). 

• Coherence variable: A G {0, 1}: coherence variable (restraining BT to possible values with 
regard to Ojl . . . Nj) 

• Time: T G [1 . . . PJ, time in the game (1 second resolution). 
Additionally, we have: 

• Opening: Op t G [openingi . . . opening m] take the various opening values (depending on 



the race), with opening labels as described in section 7.3.2 



• Last opening: Op 1 1 G [openingi ■ ■ - opening^, opening value of the previous time step 
(this allows filtering, taking previous inference into account). 

Decomposition 

The joint distribution of our model is the following: 

P(T, BT, O l . . . 0 N , Op 1 , Op 1 ' 1 , A) (7.3) 

= p(Op'|Op^ 1 )p(Op^ 1 )p( J Br|Op*)p(r| J BT,Op t ) (7.4) 

xP(A|£T,0 I1 .. JV] )n£ 1 P(O i ) (7.5) 
This can also be seen as Figure [7. 10 



Forms 



P(Op t \Op t 1 ), we use it as a filter, so that we take into account previous inferences (com- 
pressed). We use a Dirac: 

P(Op l = op t \Op t ~ l = op' -1 ) = 1 if op 1 = op l ~ l 
P(Op* = op t \Op t ^ 1 = op' -1 ) = 0 else 

This does not prevent our model to switch predictions, it just uses the previous inference's 
posterior P(Op i_1 ) to average P(Op'). 
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Figure 7.10: Graphical model (plate notation) of the opening Bayesian model, gray variables 
are known. 



Y(Op ) is copied from one inference to another (mutated from P(Op )). The first 
P(Op'~ 1 ) is bootstrapped with the uniform distribution. We could also use a prior on 



openings in the given match-up, which is directly shown in Tables 7.1 and 7.3 



P(BT\Op t = op) = Categorical(\BT\ ,p op ) is learned from the labeled replays. P(BT\Op t ) 
are M (^{openings}) different histogram over the values of BT. 

P(Oj 6 n jv]) is unspecified, we put the uniform distribution. 

P(\\BuildTree, Oum) is a functional Dirac that restricts BuildTree values to the ones 
than can co-exist with the observations. As explained above in the build tree model. 



P(T\BT = bt,Op l = op) = N(fJ>bt,op,(Tbt,op) : P(T\BT,Op l ) are discrete normal distribu- 
tions ("bell shapes"). There is one bell shape over T per couple {opening, buildTree). The 
parameters of these discrete Gaussian distributions are learned from the labeled replays. 



Identification (learning) 



The learning of the V(BT\Op l ) histogram is straight forward counting of occurrences from the 
labeled replays with "add-one smoothing" (Laplace's law of succession |Jaynes[|2003j ): 



P(BT = btpp 1 = op) 



1 + count _ games (bt A op) 
\BT\ + count _ games (op) 



The learning of the P(T\BT, Op 1 ) bell shapes parameters takes into account the uncertainty 
of the couples (bt, op) for which we have few observations. This is even more important as 
observations may (will) be sparse for some values of Op 1 , as can be seen in Figure 
the previous, the normal distribution ~P(T\bt, op) begins with a high a\ t . 
fact that the first (s) observation(s) may be outlier (s). 



7.11 



As for 



This accounts for the 
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Questions 

The question that we will ask in all the benchmarks is: 

P(Op*|T = i,0[i..jv] = 0[i..jv], A = 1) 



(7.6) 



«n£iP(^)Eo P - 1 P(^|Op t - 1 )E B TP(A|i?T, 0[1 ...^)P( J Br|Op t ).P(t| J Br,Op') (7.7) 

Note that if we see ~P(BT, Time) as a plan, asking P(i?T|Op*, Time) boils down to use our "plan 
recognition" mode as a planning algorithm. This gives us a distribution on the build trees to 
follow (build orders) to achieve a given opening. 

Bayesian program 



O 
t-i 

'53 



u 

CO 

p 



u < 
'u 

CO 



T, ST, 0 1 ...0 N ,Op t ,Op t - 1 ,\ 
Decomposition 

P(T, ST, Oi . . . 0 N , Op\ Op 1 - 1 , A) = JD 
= P(Op*|Op*- 1 )P(Op*- 1 )P(5T|Op*) 

p(\\bt, o [1 ... JV] )P(r|Br, Op*) n£i P(Oi) 

Forms 

P (Op* | Op*" 1 ) = Dime (filtering) 

P(O p * = 0 p* | Op*" 1 = op*" 1 ) = 1 iff op* == op* _1 ,0 else 
P(5T|Op* = op) = Categorical{\BT\ ,p op ) 
P(\\BT, 0[i...jv]) = functional Dirac (coherence) 
P(T|BT = fo, Op* = op) = discreteN(u bt<op , a 2 bt op ) 
Identification 

P(BT = btW = op) = i^Tco^g) 
_ (/iw,op, o-w,op) = argmax^ P(T|5T = &t, Op* = op; o 2 ) 
Question 

s P(Op*|T = t,O tl _ N j = 0[!...jv], A = 1) oc Eop*- 1 Y,BT JD 



7.6.2 Results 
Metrics 

For each match-up, we ran cross-validation testing with 9/10th of the dataset used for learning 
and the remaining l/10th of the dataset used for testing. We ran tests finishing at 5, 10 and 
15 minutes to capture all kinds of openings (early to late ones). To measure the predictive 
capability of our model, we used 3 metrics: 

• the final prediction, which is the opening that is predicted at the end of the test, 

• the online twice (OT), which counts the openings that have emerged as most probable 
twice a test (so that their predominance is not due to noise), 
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Figure 7.11: P(T, BT\Op l = ReaverDrop) Values of BT are in y-axis and values of T in x-axis. 
We have sparse 1-dimensional Gaussian distributions depending on T for each value of BT. 



• the online once > 3 (003), which counts the openings that have emerged as most prob- 
able openings after 3 minutes (so that these predictions are based on really meaningful 
information) . 

After 3 minutes, a Terran player will have built his first supply depot, barracks, refinery (gas), 
and at least factory or expansion. A Zerg player would have his first overlord, zergling pool, 
extractor (gas) and most of the time his expansion and lair tech. A Protoss player would have 
his first pylon, gateway, assimilator (gas), cybernectics core, and sometimes his robotics center, 
or forge and expansion. 



Predictive power 



Table 7.8 sums up all the prediction probabilities (scores) for the most probable opening accord- 



ing to our model (compared to the replay label) in all the match-ups with both labeling of the 



game logs. The first line should be read as: in the Protoss vs Protoss (PvP) match-up", with 



the rule-based openings labels (Weber's labels), the most probable opening * (in the Op values) 
at 5 minutes ("final") is 65% correct. The proportion of times that the most probable opening 
twice ("OT") in the firsts 5 minutes period was the real one (the game label for this player) is 
53%. The proportion of times that the most probable opening after 3 minutes ("003") and in 
the firsts 5 minutes periods was the real one is 0.59%. Then the other columns show the same 
metrics for 10 and 15 minutes periods, and then the same with our probabilistic clustering. 

Note that when an opening is mispredicted, the distribution on openings is often not 
P '(badopening) = 1,P (others) = 0 and that we can extract some value out of these distri- 
butions (see the bot's strategy adaptation in chapter [8]). Also, we observed that P(Opening = 
unknown) > P (others) is often a case of misprediction: our bot uses the next prediction in this 
case. 
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Figure 7.12: Evolution of P(Opening) with increasing observations in a TvP match-up, with 
Weber's labeling on top, our labeling on the bottom. The x-axis corresponds to the construction 
of buildings. The interpolation was obtained by fitting a second order polynomial. 



Figure 7.12 shows the evolutions of the distribution P(Opening) during a game for Weber's 

and rule- 



and our labeling or the openings. We can see that in this game, their build tree 



based labeling (top plot) enabled the model to converge faster towards FastDropship. With our 
labeling (bottom plot), the corresponding drop opening peaked earlier (5th building vs 6th with 
their labeling) but it was then eclipsed by two_facto (while staying good second) until later 
with another key observation (11th building). This may due to unorthodox timings of the drop 
opening, probably because the player was delayed (by a rush, or changed his mind), and that 
the two _ facto opening has a larger (more robust to delays) support in the dataset. 



Robustness to noise 

Figure [7. 13 shows the resistance of our model to noise. We randomly removed some observations 
(buildings, attributes), from 1 to 15, knowing that for Protoss and Terran we use 16 buildings 
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observations and 17 for Zerg. We think that our model copes well with noise because it backtracks 
unseen observations: for instance if we have only the core observation, it will work with build 
trees containing core that will passively infer unseen pylon and gate. 





Figure 7.13: Two extreme evolutions of the 3 probabilities of opening recognition with increasing 
noise (15 missing attributes/observations/buildings correspond to 93.75% missing information 
for Protoss and Terran openings prediction and 88.23% of missing attributes for Zerg openings 
prediction). Zerg opening prediction probability on top, Protoss bottom. 
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Computational performances 



The first iteration of this model was not making use of the structure imposed by the game in the 
form of "possible build trees" and was at best very slow, at worst intractable without sampling. 



With the model presented here, the performances are ready for production as shown in Table 7.6 
The memory footprint is around 3.5Mb on a 64bits machine. Learning computation time is linear 
in the number of games logs events (O(N) with N observations), which are bounded, so it is 
linear in the number of game logs. It can be serialized and done only once when the dataset 
changes. The prediction computation corresponds to the sum in the question (III.B.5) and so 
its computational complexity is in 0(N ■ M) with N build trees and M possible observations, 
as M « N, we can consider it linear in the number of build trees (values of BT). 



Table 7.6: Extremes of computation time values (in seconds, Core 2 Duo 2.8Ghz) 



Race 


Nb Games 


Learning time 


Inference \jl 


Inference a 1 


T (max) 


1036 


0.197844 


0.0360234 


0.00892601 


T (Terran) 


567 


0.110019 


0.030129 


0.00738386 


P (Protoss) 


1021 


0.13513 


0.0164457 


0.00370478 


P (Protoss) 


542 


0.056275 


0.00940027 


0.00188217 


Z (Zerg) 


1028 


0.143851 


0.0150968 


0.00334057 


Z (Zerg) 


896 


0.089014 


0.00796715 


0.00123551 



Strengths and weaknesses of StarCraft openings 



We also proceeded to analyze the strengths and weaknesses of openings against each oth- 
ers. For that, we labeled the dataset with openings and then learned the V(Win = 
truelOppi ^jOppi r2 ) probability table with Laplace's rule of succession. As can be seen 



in Table |7.1[ not all openings are used for one race in each of its 3 match-ups". Table 7.6.2 
shows some parts of this P(Win = true\Op playerl , Op player2 ) ratios of wins for openings against 
each others. This analysis can serve the purpose of choosing the right opening as soon as the 
opponent's opening was inferred. 



Zerg Protoss 


two gates fast dt reaver drop corsair nony 


speedlings 
lurkers 
fast mutas 


0.417 0.75 NED NED 0.5 
NED 0.493 NED 0.445 0.533 
NED 0.506 0.5 0.526 0.532 



Terran Protoss 


fast dt reaver drop corsair nony 


two facto 
rax fe 


0.552 0.477 NED 0.578 
0.579 0.478 0.364 0.584 



Table 7.7: Opening/Strategies labels counted for victories against each others for the PvZ (top, 
on 1408 games) and PvT (bottom, on 1657 games) match-ups. NED stands for Not Enough 
Data to conclude a preference/discrepancy towards one opening. The results should be read as 
win rates of columns openings vs lines openings. 
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7.6.3 Possible uses 



We recall that we used this model for opening prediction, as a proxy for timing attacks and 
aggressiveness. It can also be used: 

• for build tree suggestion when wanting to achieve a particular opening. Particularly one 
does not have to encode all the openings into a finite state machine: simply train this 
model and then ask P(BT\time, opening, A = 1) to have a distribution on the build trees 
that generally are used to achieve this opening. 

• as a commentary assistant AI. In the StarCraft and StarCraft 2 communities, there are 
a lot of progamers tournaments that are commented and we could provide a tool for 
commentators to estimate the probabilities of different openings or technology paths. As 
in commented poker matches, where the probabilities of different hands are drawn on 
screen for the spectators, we could display the probabilities of openings. In such a setup 
we could use more features as the observers and commentators can see everything that 
happens (upgrades, units) and we limited ourselves to "key" buildings in the work presented 
here. 



Possible improvements 



First, our prediction model can be upgraded to explicitly store transitions between t and t + 1 
(or t- 1 and t) for openings (Op) and for |build trees] " (BT). The fact is that P(BT t+l \BT t ) 
will be very sparse, so to efficiently learn something (instead of a sparse probability table) 
we have to consider a smoothing over the values of BT, perhaps with the distances men- 
tioned in section 7.5.2 If we can learn P(BT t+1 \BT t ), it would perhaps increase the results of 
P(Opening\Observations), and it almost surely would increase P(BuildTree t+1 {Observations), 
which is important for late game predictions. 

Incorporating P(Op t_l ) priors per match-up (from Table 7.1) would lead to better results, 



but it would seem like overfitting to us: particularly because we train our robot on games played 
by humans whereas we have to play against robots in competitions. 

Clearly, some match-ups are handled better, either in the replays labeling part and/or in 
the prediction part. Replays could be labeled by humans and we would do supervised learning 
then. Or they could be labeled by a combination of rules (as in |Weber and Mateas| |2009| ) 
and statistical analysis (as the method presented here). Finally, the replays could be labeled 
by match-up dependent openings (as there are different openings usages by match-ups", see 



Table 7.1 ), instead of race dependent openings currently. The labels could show either the two 



parts of the opening (early and late developments) or the game time at which the label is the 
most relevant, as openings are often bimodal ("fast expand into mutas", "corsairs into reaver", 
etc.). 

Finally, a hard problem is detecting the "fake" builds of very highly skilled players. Indeed, 
some progamers have build orders which purpose are to fool the opponent into thinking that they 
are performing opening A while they are doing B. For instance they could "take early gas" leading 
the opponent to think they are going to do tech units, not gather gas and perform an early rush 
instead. We think that this can be handled by our model by changing P(Opening\LastOpening) 
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by P(Opening\LastOpening, LastObservations) and adapting the influence of the last predic- 
tion with regard to the last observations (i.e., we think we can learn some "fake" label on replays). 
If a player seem on track to perform a given opening but fails to deliver the key characteristic 
(heavy investment, timing attack...) of the opening, this may be a fake. 



7.7 Army composition 

We reuse the predictions on the build tre"e| " (P(BT)) directly for the tech tree] " (P(TT)) (for the 
enemy, so ETT) by estimating BT as presented above and simply adding the tech tree additional 
features (upgrades, researches) that we already sawj^] You can just assume that BT = TT, or 
refer to section [7.3.11 



7.7.1 Bayesian model 

In this model, we assume that we have some incomplete knowledge of the opponent's tech tree 
and a quite complete knowledge of his army composition. We want to know what units we should 
build now, to adapt our army to their, while staying coherent with our own tech tree and tactics 
constraints. To that end, we reduce armies to mixtures of clusters that could have generated a 
given composition. In this lower dimension (usually between 5 to 15 mixture components), we 
can reason about which mixture of clusters the opponent is probably going to have, according 
to his current mixture components and his tech tree. As we learned how to pair compositions 
strengths and weaknesses, we can adapt to this "future mixture". 



Variables 



TT* is a tech tree" variable, at time t. TT £ {®,{buildingi},{building2},{buildingi A 
building2} , . . . }: all the possible building trees for the given race. We just want to know 
what unit types we can produce and not recreate the whole technological state. TT has 



V possible values (V is the number different tech trees" and depends on the faction). 

ETT 1 the enemy tech tree, same as above but for the enemy. ETT has V' possible values. 

EC t,t+1 G {enemy's race's clusters} the enemy's cluster (EC) estimated at t from their 
units that we saw, and estimated at t + 1 from their (estimated distribution on) tech 
trees (ETT) and previous EC 1 . EC t , EC t+1 each have K' values (number of mixtures 
components). 

Cftm f ^ {our race's clusters}, our army cluster (C), both wanted (Ct for tactics,C c for 
counter, C m for merge) and decided (Cf for final) at £ + 1. Ct, C c , C m , Cf have K values 
(K units clusters for us). We infer C c the "counter" cluster for (adaptation to) EC t+1 . 
Ct (a distribution Ct) comes from the needs of specific units by the tactical model, we 
merge Ct and C c in C m (merge). The final clusters (Cf) corresponds to what is coherent 
with our tech tree (TT) so use a coherence variable (A) to make C m and Cf coherent. See 



Figure 7.14 



7 We could also modify the models above and incorporate tech tree features but it is of little benefit. Also, 
BT contains duplicates of buildings and is it easier to downgrade to TT than to estimate the openings from TT 
instead of BT. 
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• U t+1 G ([0, 1] . . . [0, 1]), our N dimensional unit types (U) proportions at time t+1, i.e. 
U t+1 G [0,1]^. For instance, an army with equal numbers of zealots and dragoons (and 
nothing else) is represented as {U zea iot = 0.5, Udragoon = 0.5, \/ut ^ zealot\dragoon U u t = 
0.0}, i.e. U = (0.5,0.5,0, ... ,0) if zealots and dragoons are the first two components of 
the U vector. So Y2i Ui = 1 whatever the composition of the army. 

• EU l and EU t+l G ([0, 1] . . . [0, 1]), the M enemy units types (EU) at time t and t + 1, i.e. 
EU t G [0, 1] M . Same as above, but for the enemy and at two different times (t and t + 

• A G {0, 1} is a coherence variable unifying C^ 1 an d Cj +1 to possible values with regard 
to TT*. 

For tech trees (TT and ETT) values, it would be absurd to generate all the possible com- 
binations, exactly as for BT (see the previous two sections). We use our previous BT and the 
researches. This way, we counted 273 probable tech tree values for Protoss, 211 for Zerg, and 
517 for Terr an (the ordering of add-on buildings is multiplying tech trees for Terran). Should it 
happen, we can deal with unseen tech tree either by using the closest one (in set distance) or 
using an additional value of no knowledge. 



Decomposition 

The joint distribution of our model is the following: 



P(TT*, Cl +1 , C*+\ C£+\ C} +1 , ETT\ EC\ EC t+1 , U t+1 ,EU l ) (7.8) 

V(EU t \EC t )Y , {EC t \EC t+l )V(EC t+l \ETT t )V{ETT t ) (7.9) 

xP(C t c +1 \EC t+1 )P{C t t +1 )P(C t + 1 \C t t +1 ,Ci +1 ) (7.10) 

xp(a|c* +1 , c^ hl )P(c* +1 |rr*)p(rr*)P(C7 t+1 |c^ +1 ) (7.11) 



This can also be see as Figure 7.14 



Forms 



PiEU^EC 1 = ec) =N{n ec , crec 2 ), as above but with different parameters for the enemy's 
race (if it is not a mirror match- up). 

V(EC l \EC t+l = ec t+1 ) = Categorical(K',p ec t+i), P(^C t | J BC* +1 ) is a probability table 
of dimensions K' x K' resulting of the temporal dynamic between clusters, that is learned 



from the dataset with a soft Laplace's law of succession ("add one" smoothing) Jaynes 
[2003] . 

V{EC t+l \ETT l = ett) = Categorical(K' ,p ett ), P(£C m |£TT<) is a probability table 
of dimensions K' x V' resulting of the inter-dependency between some tech trees and 
clusters/mixtures. It is learned from the dataset with a soft Laplace's law of succession. 

P(ETT t ) = Categorical(V 

iPeracett) > it is a categorical distribution on V' values, i.e. 
an histogram distribution on the enemy's tech trees. It comes from the build/tech tree 



prediction model explained above (section 7.5). For us, TT ~ Categorical(V) too, and 
we know our own tech tree (TT) exactly. 
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Figure 7.14: Left: the full Gaussian mixture model for Q armies (Q battles) for the enemy, 
with K' mixtures components. Right: Graphical (plate notation) representation of the army 
composition Bayesian model, gray variables are known while for gray hatched variables we have 
distributions on their values. Ct can also be known (if a decision was taken at the tactical model 
level) . 



• V{C l c +1 \EC t+l = ec) = Categorical(K,p ec ), P(C* +1 |£C m ) is a probability table of 
dimensions K x K f , which is learned from battles with a soft Laplace's law of succession 
on victories. 

• P(C* +1 ) = Categorical(K , ptac) (histogram on the K clusters values) coming from the 
tactical model. Note that it can be degenerate: P(C* +1 = shuttle) = 1.0. It serves the 
purpose of merging tactical needs with strategic ones. 

• P(C t + 1 \C t t +1 ,C t c +1 ) = aP(Cf~ 1 ) + (l-a)P(C , *+ 1 ) with a G [0 . . . 1] the aggressiveness/ini- 
tiative parameter which can be set fixed, learned, or be variable (P(a\ situation)). If a = 1 
we only produce units wanted by the tactical model, if a = 0 we only produce units that 
adapts our army to the opponent's army composition. 

• P(A|Cj +1 , C^ 1 ) is a functional Dirac that restricts C m values to the ones that can co-exist 
with our tech tree (Cf). 

P(X = l\C f ,C m = c m ) 
= 1 iff P(C, = c m ) ^ 0 
= 0 else 

This was simply implemented as a function. This is not strictly necessary (as one could 
have P(Cf\TT, C m ) which would do the same for P(C/) = 0.0) but it allows us to have the 
same table form for P(Cf\TT) than for P(EC\ETT), should we wish to use the learned 
co-occurrences tables (with respect to the good race). 

• P(Cj. +1 |TT) is either a learned probability table, as for P{EC t+1 \ETT t ), or a functional 
Dirac distribution which tells which Cf values (cy) are compatible with the current tech 
tree TT = tt. We used this second option. A given c is compatible with the tech tree 
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(tt) if it allows for building all units present in c. For instance, tt = {pylon, gate, core} is 
compatible with a c = {/Ut/ zea , ot = 0.5, (J-u dragoon = 0.5, \/ut / zealot\dragoon fiu ut = 0.0}, 
but it is not compatible with c' = {fJ>u Tbiter = 0.1, . . . }. 

V(U t+l \C l f +l = c) = Af(fi c ,a 2 c ), the u and <r come from a Gaussian mixture model learned 



from all the battles in the dataset (see left diagram on Fig. 7.14 and section 7.4.2) 



Identification (learning) 

We learned Gaussian mixture models (GMM) with the expectation-maximization (EM) algo- 



rithm on 5 to 15 mixtures with spherical, tied, diagonal and full co- variance matrices |Pedregosa 



et al. 20111. We kept the best scoring models according to the Bayesian information criterion 



(BIC) |Schwarz 1978|. 



P([/' +1 |Cj. +1 = c) is the result of the clustering of the armies compositions (U) into C, 
and so depends from the clustering model. In our implementation, u c , o~ c are learned from 
the data through expectation maximization. One can get a good grasp on this parameters 



by looking at Figure 7.16 



It is the same for fi ec ,a ec , they are learned by EM on the dataset (see section 7.4.2). 



P(ETT t ) comes from the model described two sections above (7.5). 
P(EC t+1 = ec\ETT l = ett) = 1+p ( ec ) xcount ( ecAeft ) 



K'+count(ett) 

P(ec)xcount batt 
i<T+P(ec)countbattles (ec) 



• P(C* +1 = c\EC t+1 = ec) = 1+P jf l*pf 6 \ X cou ° tbattlc , s ( 6>ec ) ; we only count when c won against 



ec. 



• Both P(Cj. +1 |TT) and P(A|Cj. +1 , C^ 1 ) are functions and do not need identification. 
Questions 

The question that we ask to know which units to produce is: 

P(U t+1 \eu t ,tt t ,X = l) (7.12) 
oc ^ |"p(i?C <+1 |£TT*)P(£TT*) (7.13) 

ETT t ,EC t ,EC t + 1 

xPieu'lEC 1 ) Yl [P(C* +1 |£C i+1 ) (7.14) 

rit+i r,t+i n t+i n t+i 

xP(C* +1 )P(Cj +1 |«*)P(A|C* +1 ,C^ 1 )P([/ m |C} +1 )]j (7.15) 

or we can sample U t+1 from P(Cj. +1 ) or even from c^ +1 (a realization of Cj =1 ) if we ask 
P(Cy~ 1 \eu\. M ,tt t , X = 1). Note that here we do not know fully neither the value of ETT 
nor of Ct so we take the most information that we have into account (distributions). The eval- 
uation of the question is proportional to \ETT\ x \EC\ 2 x |C| 4 = V x K' 2 x K 4 . But we do 
not have to sum on the 4 types of C in practice: P(C m \Ct,C c ) is a simple linear combination 
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of vectors and P(A = l\Cf,C m ) is a linear filter function, so we just have to sum on C c and 
practical complexity is proportional to V' x K' 2 x K. As we have most often ~ 10 Gaussian 
components in our GMM, K or K' are in the order of 10 (5 to 12 in practice), while V and V' 
are between 211 and 517 as noted above. 



Bayesian program 



cS 
t-t 
bJO 
O 



o 
PQ 



u 
Q 



o 

. 

u < 

eg 

'u 

o 



Variables 

TT\ C\ +1 , C* +1 , C£+\ C l f +1 ,ETT\ EC 1 , EC t+1 , U t + 1 ,EU t 
Decomposition 

P(TT\ C t t +1 ,C t + 1 ,C t + 1 , Cp 1 ,ETT t , EC 1 , EC t+1 , U t + 1 ,EU t 
= P(EU t \EC t )P(EC t \EC t+1 )P(EC t+1 \ ETT 1 ) P(ETT t ) 
xP(C*+ 1 |i?C rt + 1 )P(^+ 1 )P(C^ 1 |C t '+ 1 ,^+ 1 ) 

xP(A|c} +1 ,c^ 1 )p(c* +1 |TT*)P(rr*)P(t/* +1 |cJ +1 ) 

Forms 

P(£*7*|£C" = ec) = A% ec , aec 2 ) 



P(£<7*|£C m = ec t+1 ) = Categorical(K', Pec t+i) 

P(EC t+1 \ETT t = ett) = Categorical(K', Pett ) 

P(ETT t ) = CategoricaliV' ,p e tt), comes from an other model 

P(C% +1 \EC t+1 = ec) = Categorical(K, Pec ) 

P(C* +1 ) = Categorical(K, Ptac ) 

P(C t + 1 \Cl + \C t + 1 ) = aP(C$ +1 ) + (1 - a)P(C^) 

P(A = l\C f , C m = c m ) = l iff P(C f = c m ) + 0, else 0 

P(C t+1 = Cf \TT = tt) = l iff c/ producible with tt, else 0 



{p{U^\C t + 1 = c)=N^ c ,al) 
Identification 

u c , o~ c learned by EM on the full dataset 
Uec, Cec learned by EM on the full dataset 
P(EC t = ec^EC^ 1 = ec' 
P(EC t+1 = ec\ETT l = ett 
P(C* +1 = c\EC t+1 = ec) = 
Question 

P(U t+1 \eu t ,tt t ,X = 1) 

« J2ETTt,EC*,EC* + i \P{EC t+1 \ETT t )P(ETT t ) 
xP(eu«|^C*) Ech WW . [P^W* 1 ) 
k xP(C t m )P(C} +1 |tt*)P(A|Cj +1 , C^+ 1 )P(C/ t+1 |C* +1 )] 



l+P(ec t )P(ec t+1 )xcount(ec f ^ec f+1 ) 
^ ~~ i^+P(ec*+ 1 )xcount(ee t + 1 ) 

1+P(ec)x count (ecAett) 
~ K'+count(ett) 
l+P(c)P(ec) X countbattles (c>ec) 
A"+P(ec)count battlos (ec) 
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7.7.2 Results 

We did not evaluate directly P(U t+1 \eu t , tt l , A = 1) the efficiency of the army compositions 
which would be produced. Indeed, it is difficult to make it independent from (at least) the 
specific micro-management of different unit types composing the army. The quality of V(ETT) 
also has to be taken into account. Instead, we evaluated the reduction of armies compositions 
(EU and U sets of variables) into clusters (EC and C variables) and the subsequent V(C C \EC) 
table. 



We use the dataset presented in section 6.4 in last chapter (tactics). It contains everything 



we need about the state and battles, units involved, units lost, winners. There are 7708 games. 
We only consider battles with about even force sizes, but that lets a sizable number on the more 
than 178,000 battles of the dataset. For each match-up, we set aside 100 test matches, and use 
the remaining of the dataset for learning. We preferred robustness to precision and thus we 
did not remove outliers: better scores can easily be achieved by considering only stereotypical 
armies/battles. Performance wise, for the biggest dataset (PvT) the learning part takes around 
100 second on a 2.8 Ghz Core 2 Duo CPU (and it is easily serializable) for 2408 games (57 seconds 
to fit and select the GMM, and 42 seconds to fill the probability tables / fit the Categorical 
distributions). The time to parse all the dataset is far larger (617 seconds with an SSD). 

Evaluation of clustering 

We will look at the predictive power of the P(C C \EC) (comprehending the reduction from U to 
C and EU to EC) for the results of battles. For each battle, we know the units involved (types 
and numbers) and we look at predicting the winner. 

Metrics 

Each battle consists in numbers of units involved for each types and each parties (the two 
players). For each battle we reduce the two armies to two Gaussian mixtures (P(C) and P(EC)). 
To benchmark our clustering method, we then used the learned V(C C \EC) to estimate the 
outcome of the battle. For that, we used battles with limited disparities (the maximum strength 
ratio of one army over the other) of 1.1 to 1.5. Note that the army which has the superior forces 
numbers has more than a linear advantage over their opponent (because of focus firing^]), so 
a disparity of 1.5 is very high. For information, there is an average of 5 battles per game at a 1.3 
disparity threshold, and the numbers of battles per game increase with the disparity threshold. 

We also made up a baseline heuristic, which uses the sum of the values of the units (see 
section 6.3.2 for a reminder) to decide which side should win. If we note v(unit) the value of a 
unit, the heuristic computes ^2 unit v(unit) for each army and predicts that the winner is the one 
with the biggest score. Of course, we recall that a random predictor would predict the result of 
the battle correctly 50% of the time. 

A summary of the main metrics is shown in Table |7.8| the first line can be read as: for a 
forces disparity of 1.1, for Protoss vs Protoss (first column), 

• considering only military units 

— the heuristic predicts the outcome of the battle correctly 63% of the time. 



8 Efnciently micro-managed, an army 1.5 times superior to their opponents can keep much more than one 
third of the units alive. 
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— the probability of a clusters mixture to win against another (P(C\EC)), without 
taking the forces sizes into account, predicts the outcome correctly 54% of the time. 

— the probability of a clusters mixture to win against another, taking also the forces 
sizes into account (¥{C\EC) x ^2 un i t v{unit)), predicts the outcome correctly 61% of 
the time. 

considering only all units involved in the battle (military units, plus static defenses and 
workers): same as above. 



And then it is given for all match-ups* (columns) and different forces disparities (lines). The 
last column sums up the means on all match-ups, with the whole army (military units plus static 
defenses and workers involved), for the three metrics. 

Also, without explicitly labeling clusters, one can apply thresholding to special units (ob- 
servers, arbiters, science vessels...) to generate more specific clusters: we did not include these 
results here (they include too much expertize/tuning) but they sometimes drastically increase 
prediction scores. 

Predictive power 

We can see that predicting battle outcomes (even with a high disparity) with "just probabilities" 
of V(C C \EC) (without taking the forces into account) gives relevant results as they are always 
above random predictions. Note that this is a very high level (abstract) view of a battle, we do 
not consider tactical positions, nor players' attention, actions, etc. Also, it is better (in average) 
to consider the heuristic with the composition of the army (probx heuristic) than to consider 
the heuristic alone, even for high forces disparity. These prediction results with "just prob.", or 
the fact that heuristic with P(C C \EC) tops the heuristic alone, are a proof that the assimilation 
of armies compositions as Gaussian mixtures of cluster works. 

Army efficiency 

Secondly, and perhaps more importantly, we can view the difference between "just prob." results 
and random guessing (50%) as the military efficiency improvement that we can (at least) 
expect from having the right army composition. Indeed, we see that for small forces disparities 
(up to 1.1 for instance), the prediction with army composition only ("just prob.": 63.2%) is 
better that the prediction with the baseline heuristic (61.7%). It means that we can expect to 
win 63.2% of the time (instead of 50%) with an (almost) equal investment if we have the right 
composition. Also, we predict 58.5% of the time the accurate result of a battle with disparity 
up to 1.5 from "just prob.": these predictions are independent of the sizes of the armies. What 
we predicted is that the player with the better army composition won (not necessarily the one 
with more or more expensive units). 

Analysis of the clustering 



Figure 7.15 shows a 2D Isomap Tenenbaum et al. 2000 projection of the battles on a small 



ZvP dataset. Isomap finds a low-dimensional non-linear embedding of the high dimensionality 
(N or N 1 dimensions, as much as unit types, the length of U or EU) space in which we represent 
armies. The fact that the embedding is quasi-isometric allows us to compare the intra- and inter- 
clusters similarities. Most clusters seem to make sense as strong, discriminating components. 
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Table 7.8: Winner prediction scores (in %) for 3 main metrics. For the left columns ("m"), we 
considered only military units. For the right columns ("ws") we also considered static defense 
and workers. The "heuristic" metric is a baseline heuristic for battle winner prediction for 
comparison using army values, while "just prob." only considers V(C C \EC) to predict the winner, 
and "prob x heuristic" balances the heuristic's predictions with X^c c EC ~P{C C \EC)¥(EC). 




Figure 7.15: 2 dimensional Isomap projection of a small dataset of battles for Zerg (vs Pro- 
toss) with most probable Gaussian mixture components as labels. The clusters are (Gaussian 
components) are labeled in colored numbers and projected in this 2 dimensional space. 



The clusters identified by the numbers 2 and 7 (in this projection) are not so discriminative, so 
they probably correspond to a classic backbone that we find in several mixtures. 

Figure 7.16 shows a parallel plot of army compositions. We removed the less frequent unit 
types to keep only the 8 most important unit types of the PvP match-up, and we display a 8 
dimensional representation of the army composition, each vertical axis represents one dimension. 
Each line (trajectory in this 8 dimensional space) represents an army composition (engaged in 
a battle) and gives the percentage^] of each of the unit types. These lines (armies) are colored 



9 scales are between 0 and a maximum value < 100%, different across unit types 
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with their most probable mixture component, which are shown in the rightmost axis. We have 
8 clusters (Gaussian mixtures components): this is not related to the 8 unit types used as the 



number of mixtures was chosen by BIC * score. Expert StarCraft players will directly recognize 
the clusters of typical armies, here are some of them: 

• Light blue corresponds to the "Reaver Drop" tactical squads, which aims are to transport 
(with the flying Shuttle) the slow Reaver (zone damage artillery) inside the opponent's 
base to cause massive economical damages. 

• Red corresponds to the "Nony" typical army that is played in PvP (lots of Dragoons, 
supported by Reaver and Shuttle). 

• Green corresponds to a High Templar and Archon-heavy army: the gas invested in such 
high tech units makes it that there are less Dragoons, completed by more Zealots (which 
cost no gas). 

• Purple corresponds to Dark Templar ("sneaky", as Dark Templars are invisible) special 
tactics (and opening). 




Dragoon 



Figure 7.16: Parallel plot of a small dataset of Protoss (vs Protoss, i.e. in the PvP match-up) 
army clusters on most important unit types (for the match-up). Each normalized vertical axis 
represents the percentage of the units of the given unit type in the army composition (we didn't 
remove outliers, so most top (tip) vertices represent 100%), except for the rightmost (framed) 
which links to the most probable GMM component. Note that several traces can (and do) go 
through the same edge. 



Analysis of dynamics 



Figure 7.17 showcases the dynamics of clusters components: P(EC t \EC t+1 , for Zerg (vs Protoss) 
for At of 2 minutes. The diagonal components correspond to those which do not change between 
t and t+1 t + 2minutes), and so it is normal that they are very high. The other components 
show the shift between clusters. For instance, the first line seventh column (in (0,6)) square 
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shows a brutal transition from the first component (0) to the seventh (6). This may be the 
production of Mutalisks 10 from a previously very low-tech army (Zerglings). 
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Figure 7.17: Dynamics of clusters: P(EC t \EC tJrl ) for Zerg, with At = 2 minutes 



Extensions 

Here, we focused on asking P(U t+1 \eu t , tt l , A = 1), and evaluated (in the absence of ground 
truth for full armies compositions) the two key components that are P(U\C) (or ~P(EU\EC)) 
and P(C C \EC). Many other questions can be asked: P(TT*|eu*) can help us adapt our tech 
tree development to the opponent's army. If we know the opponent's army composition only 
partially we can benefit of knowledge about ETT to know what is possible, but also probable, 
by asking P (EC* \ Observations). 



Mutalisks are flying units which require to unlock several technologies and thus for which player save up for 
the production while opening their tech tree. 
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7.8 Conclusion 



We contributed a probabilistic model to be able to compute the distribution over openings 
(strategies) of the opponent in a RTS game from partial and noisy observations. The bot can 
adapt to the opponent's strategy as it predicts the opening with 63 — 68% of recognition rate 
at 5 minutes and > 70% of recognition rate at 10 minutes (up to 94%), while having strong 
robustness to noise (> 50% recognition rate with 50% missing observations). It can be used in 
production due to its low CPU (and memory) footprint. 

We also contributed a semi-supervised method to label RTS game logs (replays) with open- 
ings (strategies). Both our implementations are free software and can be found online 11 We use 
this model in our StarCraft AI competition entry bot as it enables it to deal with the incomplete 
knowledge gathered from scouting. 

We presented a probabilistic model inferring the best army composition given what was 
previously seen (from replays, or previous games), integrating adaptation to the opponent with 
other constraints (tactics). One of the main advantages of this approach is to be able to deal 
natively with incomplete information, due to player's intentions, and to the fog of war in RTS. 
The army composition dimensionality reduction (clustering) can be applied to any game and 
coupled with other techniques, for instance for situation assessment in case-based planning. 
The results in battle outcome prediction (from few information) shows its situation assessment 
potential. 



11 https : //github . com/SnippyHolloW/OpeningTech/ 
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Chapter 8 



BroodwarBotQ 



Dealing with failure is easy: Work hard to improve. Success is also easy to 
handle: You've solved the wrong problem. Work hard to improve. 



Alan J. Per lis (1982) 

IN this chapter, we present some of the engineering that went in the robotic player (bot) 
implementation, which may help the comprehension of the organization and utility of the 
different chapters. We will also present the different flows of informations and how decisions are 
made during a game. Finally we will present the results of the full robotic player to various bots 
competitions. 



18.1 Code architecture! 159 

|8.2 A game walk-through| 166 

18.3 Resultel 170 



8.1 Code architecture 

Our implementation^] (BSD licensed) uses BWAP^] to get information from and to control 
StarCraft. The bot's last major revision (January 2011) consists of 23,234 lines of C++ code, 
making good use of boost libraries and BWTA (Brood War Terrain Analyzer). The learning of 
the parameters from the replays is done by separated programs, which serialize the probability 
tables and distribution parameters, later loaded by BroodwarBotQ each game. 



The global (simplified) view of the whole bot's architecture is shown in Figure 8.1 There 
are three main divisions: "Macro" (economical and production parts), "Intelligence" (everything 
information related) and "Micro" (military units control/actions). Units (workers, buildings, 
military units) control is granted through a centralized Arbitrator to "Macro" parts and Goals. 
This is a bidding system in which parts wanting a unit bid on it relatively to the importance of 
the task they will assign it. There may (should) be better systems but it works. We will now 
detail the parts which were explained in the previous three chapters. 



1 BroodwarBotQ, code and releases: http://github.com/SnippyHolloW/BroodwarBotQ 



2 BWAPI: http : //code . google . com/p/bwapi/ 
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Figure 8.1: Simple view of the code architecture of BroodwarBotQ, the most important 
interactions are shown: every piece which has responsibility for the control of units refer to the 
Arbitrator, all macro components compete for resources, all other arrows represent orders or 
important transfers of information. 



8.1.1 Units control 

As presented in chapter [5j units are controlled in a sensory-motor fashion through Bayesian 
fusion of physical or abstract influences. Units pursuing a common objective are regrouped in a 
UnitsGroup, which sets this objective for every unit it commands. This objective is generated 
for each unit from the needs of a higher-up Goal (see below) through achieve () or cancel (). 



The BayesianUnit class has different fusion modes (see section 5.3.3) set by the UnitsGroup 
depending on the Goal type and on the situation (number and types of enemies, terrain...). 
Its sensory inputs are fed by the UnitsGroup (objectives) and by the MapManager (potential 
damages, terrain) and the EUnitsFilter (enemy units). 

A derivative (child) of the BayesianUnit class is instantiated for each unit that is controlled. 



A BayesianUnit is a modal FSM* as shown in Figure 5.7 It presents a simple interface to 
move and fight (the micro() method). Some parameter can be specialized depending on the 
particularities of unit types: 

• the list of priority unit types to target (because of units attacks efficiencies as in rock/pa- 
per/scissors), 

• the reasons for which to flee (flee?()), 

• micro () itself can be specialized further if the two points above do not suffice to produce 
a skillful behavior (for very special unit types: casters). 

A call to micro () then decides of the course of actions during a fight (applying Bayesian fusion 
with the right sensory inputs when moving). 
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8.1.2 Tactical goals 

The decision that we want our AI system to make at this level is where and how to attack. This 
is reflected in the StarCraft bot as a Goal creation. Goals are interfacing high level tactical 
thinking with the steps necessary to their realization. A Goal recruits units and binds them 



under a UnitsGroup (see section 5.3.3). A Goal is an FSM" in which two states are simple 



planners (an FSM with some form of procedural autonomy), it has: 

• preconditions, for instance a drop attack needs to specify at least a transport unit (Shut- 
tle/Dropship/Overlord) and ground attack units. 

• hard states: waiting precondition, in progress, in cancel, achieved, canceled, which corre- 
sponds to the Goal advancement.. 

• and and/or or logic subgoals with: 

— a realization test 

— a proposition of action to try to realize it 

— an estimation of the "distance" to realization 

In the in progress and in cancel modes, the "plan" is a simple search in the achievable subgoals 
and their "distance" to realization. 

The tactical model can specify where to attack by setting a localized subgoal (Forma- 
tion/See/Regroup/KillSubgoal...) to the right place. It can specify how by setting the adequate 
precondition(s). It also specifies the priority of a Goal so that it has can bid on the control of 
units relatively to its importance. The GoalManager updates all Goals that were inputed and 
act as a proxy to the Arbitrator for them. 

8.1.3 Map and movements 

The MapManager keeps track of: 

• the economical and military bases (positions) of the enemy, 

• the potential damages map, which is updated with each new observations (units, spells...), 

• the "walkability" of the terrain (static terrain and buildings). 

It also provides threaded pathfinder services which are used by UnitsGroups to generate objec- 
tives waypoints when the Goal's objectives are far. This pathfinder can be asked specifically to 
avoid certain zones or tiles. 

8.1.4 Technology estimation 



The ETechEstimator does constant build tree" prediction as explained in section 7.5 and so it 



exposes the distribution on the enemy's tech tree * as a "state estimation service" to other parts 



of the bot taking building and production decision. It also performs openings * prediction during 



the first 15 minutes of the game, as explained in section 7.6 
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New enemy units observations are taken into account instantly. When we see an enemy unit 
at time t, we infer that all the prerequisites were built at least some time t' earlier according to 
the formula: t' = t — ubd — umd with ubd being the unit's building duration (depending on the 
unit type), and umd being the unit's movement duration depending on the speed of the unit's 
type, and the length of the path from its current position to the enemy's base. We can also 
observe the upgrades that the units have when we see them, and so we take that into account the 
same way. For instance, if a unit has an attack upgrade, it means that the player has the require 
building since at least the time of observation minus the duration of the upgrade research. 



The distribution on openings" computed by the ETechEstimator serves the purpose of rec- 
ognizing the short term intent of the enemy. This way, the ETechEstimator can suggest the 
production of counter measures to the opponent's strategy and special tactics. For instance, 
when the belief that the enemy is doing a "Dark Templars" opening (an opening aimed at rush- 
ing invisible technology before the time at which a standard opening reaches detector technology, 
to inflict massive damage) pass above a threshold, the ETechEstimator suggests the construction 
of turrets (Photon Cannon, static defense detectors) and Observers (mobile detectors). 

8.1.5 Enemy units filtering 

At the moment, enemy units filtering is very simple and just diffuse uniformly (with respect to 
its speed) the probability of a unit to be where it was last seen before totally forgetting about 
its position (not its existence) when it was not seen for too long. We will now shortly present 
an improvement to this enemy units tracking. 

A possible improved enemy units tracking/filtering 

We consider a simple model, without speed nor steering nor See variable for each position/unit. 
The fact that we see positions where the enemy units are is taken into account during the update. 
The idea is to have (learn) a multiple influences transition matrix on a Markov chain on top 
of the regions discretization (see chapter j6|, which can be seen as one Markov chain for each 
combination of motion- influencing factors. 

The perception of such a model would be for each unit if it is in a given region, as the 



bottom diagram in Figure B.2 From there, we can consider either map-dependent regions, 
regions uniquely identified as they are in given maps that is; or we can consider regions labeled 
by their utility. We prefer (and will explain) this second approach as it allows our model to be 
applied on unknown (never seen) maps directly and allows us to incorporate more training data. 
The regions get a number in the order in which they appear after the main base of the player 



(e.g. see region numbering in Fig. 6.2 and 8.2). 



Variables 

The full filter works on n units, each of the units having a mass in the each of the m regions. 

• Agg G {true, false}, the player's aggressiveness (are they attacking or defending?), which 
can comes from other models but can also be an output here. 

• C^Tjgjx n j 6 {unit types} the type of the ith tracked unit, with K unit types. 

• X*~h n j £ [ri . . . r m J, the region in which is the ith unit at time t — 1. 
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• ^* e [x n j G i r i ■ ■ ■ r m J, the region in which is the ith unit at time t. 
Joint Distribution 

We consider all units conditionally independent give learned parameters. (This may be too 
strong an assumption.) 



P(^,[/T 1:m ,X^ : J (8.1) 

n 

P(Agg) H [P([/T i )P(X l '- 1 )P(X*|X l '- 1 , UT U Agg)] (8.2) 

i=l 



Forms 



• P(Agg) = Binomial{p a gg) is a binomial distribution. 

• P(UTi) = Categorical(K , p u t) is categorical distribution (on unit types). 

• P(X*~ 1 ) = Categorical(m,p reg ) is a categorical distribution (on regions). 

• P(X||X J * — , UTi, Agg) are #{units _types} x \Agg\ = K x 2 different to x to matrices 
of transitions from regions to other regions depending on the type of unit i and of the 
aggressiveness of the player. (We just have around 15 unit types per race {K ~ 15) so we 
have ~ 15 x 2 different matrices for each race.) 



Identification (learning) 

P(X*|X* _1 ,UTi, Agg) is learned with a Laplace rule of succession from previous games. 
Against a given player and/or on a given map, one can use P(X*| X* -1 , UTi, Agg, Player, Map) 
and use the learned transitions matrices as priors. When a player attacks in the dataset, we can 
infer she was aggressive at the beginning of the movements of the army which attacked. When 
a player gets attacked, we can infer she was defensive at the beginning of the movements of the 
army which defended. At other times, P(Agg = true) = P(Agg = false) = 0.5. 

PP(Xj = rlXt 1 = r', UT = ut, Agg = agg) 

1 + n tT ansitions{r' -> r, ut, agg)P(agg) 

OC ~ — ; 

#{entries_to_r} + YJj=i n transitions{r' -> rj,ut,agg)P(agg) 
Note that we can also (try to) learn specific parameters of this model during games (not a 



replay, so without full information) against a given opponent with EM * to reconstruct and learn 
from the most likely state given all the partial observations. 

Update (filtering) 
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• When a unit i becomes hidden, which was seen in region r just before, we have: 

P(X* = r) = 1.0 
P(X| = rj) = 0.0 Vj iff rj / r 

• For all regions r that we see "totally" (above a threshold of a percentage of the total area), 
we set P(-X| = r) = 0.0 and redistribute their probability mass to hidden (or still partially 
hidden) regions (algorithm |6J): 

Algorithm 6 Culling/updating algorithm for filtering visible regions 
for all i G {enemy _units} do 
s <- 0.0 

for all r G {visible _regions} do 

s «- s + P(X*~ 1 = r) 

P(X* -1 = r) = 0.0 
end for 

total ^ ^i^t 1 = r 3 ) 

for all r £ {^visible _regions} do 

P(X* = r) <- P(X* = r) + ^ x P^ 1 = r) 
end for 
end for 



• For all the other cases, we have: 

m 

P(X| = r) = J2 P(** -1 = rf)P(^i = r \ X t' = r'j, UT U Agg) 
j'=i 

• When we want to infer where the n enemy units are, we ask: 

P{Xl :n \UT 1:n = ut 1:n ) 

n 

oc ^ P(Agg) ]J P(uU) ]T Pp^" 1 , ut h Agg) 

Agg i=l xj" 1 

true n m 

oc ^ P(4 55 ) [] P(C/T i = «ti) P(*?|-X? _1 = ^ uti, A 55 ) 

Agg=false i=l j=l 

• When we want to infer the aggressiveness of the enemy (from the troupes' movements that 
we have seen), we ask: 

P(Agg\UT 1:n = ut 1:n ) 

n 

oc P(A^)IJp(uti) ^ P^-^^P^IX*- 1 ,^,,^) 
i=i x t-i x t 



164 




agressive 




defensive 




O 



resources 



Player 1 bases 
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Figure 8.2: Example of the units filter transition matrices on a simple map (same as in Fig. 6.2) 



for aggressive and defensive sets of parameters for a given unit type. The thickness of the arrows 
is proportional to the transition probability between regions. 



Discussion 



Offline learning of matrices (as shown in Fig. 8.2) of aggressive and defensive probable region 
transitions (for a given unit type) should be simple maximum likelihood (or add-one smoothing) 
on what happens in replays. With online learning (taking previous offline learned parameters 
as priors), we could learn preferences of a given opponent. 



By considering a particle filter Thrun, 2002 , we could consider a finer model in which we 



deal with positions of units (in pixels or walk tiles or build tiles) directly, but there are some 
drawbacks: 

• The data to learn (offline or online) is sparse as there are several versions of the same map 
and the combinatorics of start positions (2 occupied start positions on 4 possible most of 
the time). This would need any form of spatial abstraction anyway, like distance to most 
walked traces. 

• Computation cost to track ~ 40 units makes it so that the particle sampling numbers 
should be low to stay real-time. Or that one should abandon the motion model for a 



Kafman filter Kalman 1960 



The advantages are not so strong: 

• (hopefully) more precision in units position estimation, 

• possibilities to have more complex motion models (updates/culling of the filter), 
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• differentiation between trajectories of ground and flying units, 

• differentiation between different trajectories of attacks inside large regions. 

We would like to implement our regions-based enemy units filtering in the future, mainly for 
more accurate and earlier tactical prediction (perhaps even drop tactics interception) and better 
tactical decision making. 



8.2 A game walk-through 

We will now show key moments of the game, as was done with a human-played game in sec- 



tion 



4.1.2 but from the bot's point of view (with debug output). 




Figure 8.3: A full screen capture of some debug output of economical parts of BroodwarBotQ. 
The orange text at the top left shows the minerals/minute and gas/minute rates as well as the 
resources reserved for planned buildings and additional supply. The teal (light blue) rectangle 
at the bottom of the game's view shows the buildings that will be constructed (and their future 
tentative positions). The big transparent (translucent) blue ellipses show the Pylons construction 
coverage. The yellow rectangles (with numbers in their top left corners) show the future buildings 
planned positions. 



First, Figure [873] shows some economical elements of BBQ. The yellow rectangles show future 
buildings placements. In the light blue (teal color) rectangle are the next buildings which are 
planned for construction with their future positions. Here, the Protoss Pylon was just added to 
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the construction plan as our Producer estimated that we will be supply" blocked in 28 seconds 



at the current production rate. This is noted by the "we need another pylon: prevision supply 84 
in 28 sec" line, as supply is shown doubled (for programmatic reasons, so that supply is always 



an integer): 84 is 42, and we have a current max supply" of 41 (top right corner). 



Our buildings placer make it sure that there is always a path around our buildings blocks with 
a flow algorithm presented in the appendix in algorithm [9] and Figure B.7 It is particularly 
important to be able to circulated in the base, and that newly produced units are not stuck 



when they are produced (as they could is there was a convex enclosure as in Fig. B.7). Buildings 
cannot be placed anywhere on the map, even more so for Protoss buildings, as most of them 
need to be build under Pylon coverage (the plain blue ellipses in the screenshot). At the same 
time, there are some (hard) tactical requirements for the placement of defensive buildings^] 
At the beginning of the game, we have no idea about what the opponent is doing and thus our 



belief about their opening equals the prior (here, we set the priori to be uniform, see section 7.6.3 
for how we could set it otherwise) we send a worker unit to "scout" the enemy's base both to 



know where it is and what the enemy is up to. Figure 8.4 shows when we first arrive at the 
opponent's base in a case in which we have a strong evidence of what the opponent is doing 
and so our beliefs are heavily favoring the "Fast Legs" opening (in the blue rectangle on the 
right). We can see that with more information (the bottom picture) we make an even stronger 
prediction. 

If our bot does not have to defend an early rush by the opponent, it chooses to do a "push" 
(a powerful attack towards the front of the opponent's base) once it has a sufficient amount of 



military units, while it expands" (take a second base) or opens up its tech tree". The first push 



is depicted in Figure 8.5 



• First (top), we regroup our forces in front of the opponent's base with a formation SubGoal. 

• By trying to enter the opponent's base (second screenshot), our units have to fight they 
way through. 

• The bottom left picture shows the distribution on P(Diri) for the possible movements 
directions of one of the units at the top. We can see that it wants to avoid collision with 
allied units while going towards its target. 

• The bottom right picture shows our units "kiting back" (retreating while fighting) to avoid 
being exposed in the ramp up the cliff (right part of the image). 



BroodwarBotQ then goes up the ramp and destroys the base of the built-in AI. 



3 A good benchmark for buildings positioning would be to test if an AI can perform a "Forge expand" which 
consists in blocking all ground paths with buildings and protecting against the early rushes with judiciously 
placed Photon Cannons, without any military unit. 
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Figure 8.4: Crops of screenshots of the first scouting (discovery) of an opponent's base in a 
Protoss vs Protoss game. The buildings shown here are the ones of the opponent. The yellow 
squares represent the pathfinding output. On the bottom, in the blue rectangle, are displayed 
the possible openings for the Protoss opponent and their respective probabilities (in percentage) 
according what we have seen. The top picture was captured a few seconds before the right one 
and thus we had less information about the opponent's buildings (the upper right part is black 
because of the fog of war). 
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Figure 8.5: Crops of screenshots of an attack towards a Protoss opponent. The first screenshot 
(top) shows the units arriving at their formation SubGoal objectives (purple disks), the second 
shows the switch to the fight mode (for ground units) with the first enemy units appearing on 
the right. The third (bottom left) and fourth (bottom right) screenshots show the battle as 
it happens. The small squares (white to blue) show the attraction of one unit for its possible 
directions (P(Din) Vi): the whiter it is, the higher the probability to go there. 
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8.3 Results 



BroodwarBotQ (BBQ) consistently beats the built-in StarCraft: Broodwar to a point that 
the original built-in AI is only used to test the stability of our bot, but not as a sparing/training 
partner. 

BroodwarBotQ took part in the Artificial Intelligence and Interactive Digital Entertainment 
(AAAI AIIDE) 2011 StarCraft AI competition. It got 67 games counted as "crashes" on 360 
games because of a misinterpretation of rules on the first frame^J which is not the real unstability 
of the bot (~ 0,75% as seen in B.8). The results of AIIDE are in Table 8.1 



bot 



refs 



win rate 



notes 



Sky net 

UAlbertaBot 

Aiur 

ItayUndermind 

EISBot 

SPAR 

Undermind 

Nova 

BroodwarBotQ 

BTHAI 

Cromulent 

bigbrother 

Quorum 



Protoss 

Protoss 

Protoss 

Zerg 

Protoss 

Protoss 

Terran 

Terran 

Protoss 

Zerg 

Terran 

Zerg 

Terran 



Churchill and Buro 2011 



Weber et al. 


2010b 


a 


Kabanza et £ 


1. 


201 


) 



Perez and Villar 



2011 



Hagelback and Johansson 2009 



0.889 
0.794 
0.703 
0.658 
0.606 
0.539 
0.517 
0.475 
0.328 
0.319 
0.300 
0.278 
0.094 



openings depending on opponent's race 
always 2-Gates opening 
robust and stochastic strategies 
6-pooling 

2-Gates or Dragoons opening 

uses Dark Templars 

Barracks units 

robust play 

adapts to the opening 

Lurkers opening 

Factory units 

Hydralisks and Lurkers 

expands and produce Factory units 



Table 8.1: Result table for the AIIDE 2011 StarCraft AI competition with 360 games played by 
each bot. 



Also in 2011 was the Computational Intelligence and Games (IEEE CIG) 2011 StarCraft AI 
competition. This competition had 10 entries^] and BroodwarBotQ placed 4th with a little luck 



of seed (it did not have to play against Aiur). The finals are depicted in Table 8.2 



bot 


race 


crashes 


games 


wins 


Skynet 


Protoss 


0 


30 


26 


UAlbertaBot 


Protoss 


0 


30 


22 


XelnagcQ 


Protoss 


3 


30 


11 


B r o o d war B ot Q 


Protoss 


2 


30 


1 



Table 8.2: Result table of the finals for the CIG 2011 StarCraft AI competition (10 entries) 



Finally, there is a continuous ladder in which BroodwarBotQ (last updated January 2012) 
is ranked between 7 and 9 (without counting duplicates, there are ~ 20 different bots) and is 
on-par with EISBot [Weber et all, |2010b| a] as can be seen in Figure [BT8] (February 2012 ladder 



4 The only losses that our bot suffers against the built-in AI are against Zerg when the built-in AI does a 
quick zergling rush attack ("6 pool") on small maps. Human players who successfully counter this have a good 
micro- management of workers as well as an efficient replanning of the first buildings (which is not BBQ's case). 

the additional terrain analysis was not serialized and taking more than one minute on the frame 0, which 
has no special tolerance as opposed as the year before; 

6 BroodwarBotQ https://github.com/SnippyHolloW/BroodwarBotQ, BTHAI http://code.google.eom/p/ 
bthai/ , AIUR http : / /code . google . com/p/ aiurproject/ , LSAI http: //cs . laf ayette . edu/~taylorm EvoBot, 
Protoss Beast Jelly http : //wwbwai . sourcef orge .net / , Xelnaga (modified AIUR), Skynet http: / / code . google . 



com/p/skynetbot/ Nova htt p: //nova .wolf work .com/| UAlbertaBot 
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ranking). We consider that this rating is honorable, particularly considering our approach and 
the amount of engineering that went in the other bots. Note that the ladder and all previous 
competitions do not allow to record anything (learn) about the opponents. 

We want to give a quick analysis of the performance of BroodwatBotQ. We did not specialize 
the bot for one (and only one) strategy and type of tactics, which makes our tactics less optimized 
than other concurrents higher up in the ladder. Also, our bot tries to adapts its strategy, which 
has two drawbacks: 

• our strategy prediction model's parameters were learned from human-played games, which 
differ from bot's games. 

• some of the decision-making (particularly with regard to arbitrage of resources) is not 
probabilistic and interfaces badly with the predictions and suggestions. For instance, 
when we predict a "Dark Templars" opening, the ETechEstimator suggests building static 



and mobile detectors, both may be out of our current tech tree": how do we allocate our 
(limited) resources and plan the constructions, while taking into consideration the current 
building and production plan? This is not solved currently in a unified manner. 
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Chapter 9 

Conclusion 



Man's last mind paused before fusion, looking over a space that included nothing 
but the dregs of one last dark star and nothing besides but incredibly thin matter, 
agitated randomly by the tag ends of heat wearing out, asymptotically, to the absolute 
zero. 

Man said, 'AC, is this the end? Can this chaos not be reversed into the Universe 
once more? Can that not be done?" 

AC said, "THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL 
ANSWER." 

Isaac Asimov (The Last Question, 1956) 



9.1 Contributions summary 

We classified the problems raised by game AI, and in particular by RTS AI. We showed how the 
complexity of modern video games makes it so that game AI systems can only be incompletely 
specified comparatively to all the states the player can put the game in. This leads to uncertainty 
about our model, but also about the model of the opponent. Additionally, video games are often 
partially observable, sometimes stochastic, and most of them require motor skills (quick and 
precise hands control), which will introduce randomness in the outcomes of player's actions. 
We chose to deal with incompleteness by transforming it into uncertainty about our reasoning 
model. We bind all these sources of uncertainty in Bayesian models. 

Our contributions about reducing the complexity of specifying and controlling game AI 
systems are: 

• In chapter [5] we produced reactive, decentralized, multi-agent control by transforming 
the incompleteness about allied units intentions into uncertainty of their future locations. 



This can be viewed as an extension of Bayesian robot programming ILebeltel et al. , 2004 



in a multi-agent setting. Instead of specifying a distribution on the possible directions 
knowing the sensory inputs, we specified the sensor distribution (independently of each 
other sensors) knowing the direction (P(Sensor\Direction)). This approach, called inverse 
programming, can be viewed as "instead of specifying the states and their transitions based 
on sensors (an FSM), we specifying what the sensors should be when we are in a given 
state", reverting some of the burden of specifying behaviors, and in a probabilistic setting. 
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This is an extension of the work on inverse programming for Unreal Tournament avatars 



Le Hy et al. , 2004 . Combined, these contributions lead to real-time micro-management 



behaviors for StarCraft, which achieved very satisfying results (ex- aequo with the best 



micro-management AI of AIIDE 2010 competition), and were published in Synnaeve and 



Bessiere 2011a 



By going up in the ladder of abstraction (strategy & tactics), we were able to exploit 
what we previously called vertical continuity (see section 2,8. 1[ ) through hierarchical mod- 
els. About strategy (chapter [7|, from low- level observations, we produced build trees 



(section 7.5 [Synnaeve and Bessiere 201 1] ) , and built upon that to infer openings 
tion 



sec- 



7.6 |Synnaeve and Bessiere 2011b| ) and to constrain the inference on the opponent's 



army composition (section 7.7). Tactics (chapter [6]) also make good use of the prediction 



on the opponent's tech tree*. When used in decision-making, our models are even more 
constrained, because we have full knowledge of our state instead of distributions. 

We also took advantage of actions sequencing, previously called horizontal continuity (see 
section 2.8.1 ), by assessing that, often, things which are done should not be undone (at least 



not immediately) and that some strategic and tactical steps are prerequisite of intended 
steps. At the strategic level, the distribution on build trees" is time-dependent: the 



sequencing is encoded in the learned discrete Gaussian distributions (section |7.5| [Synnaeve 
and Bessiere} |2011| ). The openings are filtered on previous inferences with a first order 



Markovian assumption (section 7.6 Synnaeve and Bessiere, 2011b| ), i.e. the value at 
time t is dependent on the value at time t — 1. The army composition model (section 7.7) 
makes use of temporal continuity to adapt the player's army to the opponent's future army 
composition. Micro-management assumes an uncertain linear interpolation of future units 
position resulting of trajectory continuity (instead of considering all possible positions), as 



explained in subsection 5.3.1 



We used machine learning to help specifying our models, both used for prediction (opponent 
modeling) and for decision-making. We exploited different datasets for mainly two separate 
objectives: 

• We produced relevant abstract models thanks to learning. For the labeling replays (allow- 
ing for supervised learning of our full openings" prediction model), we used semi-supervised 
(by selecting features and a scoring function) GMM K clustering of replays, as presented 
in section 7.4 [Synnaeve and Bessiere 2011b|. In order to reason qualitatively and quan- 



titatively about armies composition with a tractable model, we applied GMM to armies 



unit types percentages to find the different composing components (section 7.7). For tac- 



tics, we used heuristics (see subsection 6.3.2 ) for the evaluation of the regions, whose bias 



(incompleteness) the model was adapted to by learning. 

We learned the parameters of our models from human-played games datasets. At the 
strategic level, we learned the time-dependent distributions on the build trees and the 



co-occurrence of openings with build trees in sections 7.5 and |7.6[ respectively published 
as |Synnaeve and Bessiere 2011 , |Synnaeve and Bessiere 2011b| . We were also able to 



study the strengths and weaknesses of openings this way (subsection 7.6.2 and we looked 
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at the dynamics or army compositions (section 7.7). For tactics (see chapter [6]) , we learned 



the co-occurrences of attacks with regions tactical properties. 

Finally, we produced a StarCraft bot (chapter [8]), which ranked 9th (on 13) and 4th (on 10), 
respectively at the AIIDE 2011 and CIG 2011 competitions. It is about 8th (on ~ 20) on an 



independent ladder containing all bots submitted to competitions (see Fig. B.8). 



9.2 Perspectives 

We will now present interesting perspectives and future work by following our hierarchical de- 
composition of the domain. Note that there are bridges between the levels presented here. In 
particular, having multi-scale reasoning seems necessary to produce the best strategies. For 



instance, no current AI is able to work out a "fast expand K " (expand before any military pro- 
duction) strategy by itself, in which it protects against early rushes by a smart positioning of 
buildings, and it performs temporal reasoning about when the opponent is first a threat. This 
kind of reasoning encompasses micro- management level reasoning (about the opponent units), 
with tactical reasoning (of where and when), buildings positioning, and economical and produc- 
tion planning. 

9.2.1 Micro-management 
Reactive behaviors 



An improvement (explained in subsection 5.6.1) over our existing model usage would consist 



in using the distributions on directions (P(Dir)) for each units to make a centralized decision 
about which units should go where. This would allow for coordinated movements while retaining 
the tractability of a decentralized model: the cost for units to compute their distributions on 
directions (P(Dir)) is the same as in the current model, and there are methods to select the 
movements for each unit which are linear in the number of units (for instance maximizing the 
probability for the group, i.e. for the sum of the movements). 

For the problem of avoiding local optima "trapping", we proposed a "trailing pheromones 



repulsion" approach in subsection 5.6.1 (see Fig. 5.14), but other (adaptive) pathfinding ap- 
proaches can be considered. 



Parameters identifications 

Furthermore, the identification of the probability distributions of the sensors knowing the di- 
rections (P (Sensor \ Direction)) is the main point of possible improvements. In the industry, 
behavior could be authored by game designers equipped with an appropriate interface (with 
"sliders") to the model's parameters. As a competitive approach, reinforcement leaning or evo- 
lutionary learning of the probability tables (or probability distributions' parameters) seems the 
best choice. The two main problems are: 

• types and/or levels of opponents: as we cannot assume optimal play from the opponents 
(at least not for large scale battles), the styles and types of the opponents' control will 
matter for the learning. 
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differences in situations: as there are several types of micro-management situations, we 
have to choose the granularity of learning settings (battles) and how we recognize them in- 
game. We could consider the continuum of situations, and use Bayesian fusion of posteriors 
from models learned in discrete contexts. 



As the domain (StarCraft) is large, distributions have to be efficiently parametrized (normal, 
log-normal, exponential distributions should fit our problem) . The two main approaches to learn 
these sensors distributions would be: 



Concurrent hierarchical reinforcement learning ( [Martin et al. 2005 showed how it can 
work in Wargus). 



Co-evolving control policies by playing them against each others (| Miles et al. 2007, Avery 



et al. 2009 presented a related work with influence maps). 



9.2.2 Tactics 



We explained the possible improvements around our model in subsection 6.7.2 The three most 
interesting research directions for tactics would be: 



Improve tactical state estimation, so that both our tactical decision-making and tactical 
prediction benefit from it. A first step would be to use a (dynamic) filtering on enemy 
units. We proposed a simpler units filtering model based on the decomposition of the map 
in regions in section [8. 1.5| 

Use our learned parameters as bootstrap ("prior") and keep on learning against a given 
opponent and/or on a given map. We should count how often and in which circumstances 
an attack, which should be successful, fails. It should even be done during a given game 
(as human players do). This may be seen as an exploration-exploitation trade-off in which 



our robotic player wants to minimize its regret for which multi-armed bandits Kuleshov 



and Precup, 2000| are a good fit. 



Tactical assault generation, so that we do not have to hard-code the tactical goals be- 



haviors. The definition of tactics used in Ponsen et al. 2006 is not exactly matching 



ours, but they evolved some strategic and tactical decision elements (evolving knowledge 
bases for |CBR ") in Wargus. However, we are still far from script-independent tactics (in 



StarCraft). Being able to infer the necessary steps to carry out a Drop (request a Drop- 
ship and military units, put them in the Dropship at some location A and drop them at 
location B to attack location C, retreat if necessary) attack would be a good benchmark 
for tactics generation. 

9.2.3 Strategy 

Higher- level parameters as variables 

Strategy is a vast subject and impacts tactics very much. There are several "strategic dimen- 
sions", but we can stick to the two strategy axes: aggressiveness (initiative), and economy/tech- 
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nology/production balance. These dimensions are tightly coupled inside a strategical plan, as 
putting all our resources in economy at time t is not aimed at attacking at time t + 1. Likewise, 
putting all our resources in military production at time t is a mistake if we do not intend to be 
aggressive at time t + 1. However, there are several combinations of values along these dimen- 
sions, which lead to different strategies. We detailed these higher order views of the strategy in 
section 17.11 

In our models (tactics Sz army composition), the aggressiveness is a parameter. Instead these 
two strategic dimensions could be encoded in variables: 

• A £ {true, false} for the aggressiveness/initiative, which will influence tactical (When do 
we attack?) and strategic models (How much do we adapt and how much do we follow 
our initiative? How do we adapt?). 

• ETP £ {eco, tech, prod} which will arbitrate how we balance our resources between eco- 
nomic expansion, technology advances and units production. 

A possibility is yet again to try and use a multi-armed bandit acting on this two variables (A 
and ETP), which (hyper-)parametrize all subsequent models. At this level, there is a lot of 
context to be taken into account, and so we should specifically consider contextual bandits. 

Another interesting perspective would be to use a hierarchy (from strategy to tactics) of 
reinforcement learning of (all) the bot's parameters, which can be seen as learning the degrees 



of liberty (of the whole bot) one by one as in Baranes and Oudeyer, 2009|. A more realistic 



task is to learn only these parameters that we cannot easily learn from datasets or correspond 
to abstract strategical and tactical thinking (like A and ETP). 

Production &: construction planning 

A part of strategy that we did not study here is production planning, it encompasses planning 
the use (and future collection) of resources, the construction of buildings and the production of 
units. Our bot uses a simple search, some other bots have more optimized build-order search, 
for instance UAlbertaBot uses a build-order abstraction with depth-first branch and bound 



Churchill and Buro |2011|. Planning can be considered (and have been, for instance Bartheye 



and Jacopin 2009| ), but it needs to be efficient enough to re-plan often. Also, it could be 



interesting to interface the plan with the uncertainty about the opponent's state (Will we be 
attacked in the next minute? What is the opponent's technology?). 

A naive probabilistic planning approach would be to plan short sequences of constructions 
as "bricks" of a full production plan, and assign them probability variables which will depend 
on the beliefs on the opponent's states, and replan online depending on the bricks probabilities. 
For instance, if we are in a state with a Protoss Cybernetics Core (and other lower technology 
buildings), we may have a sequence "Protoss Robotics Facility —> Protoss Observatory" (which 
is not mutually exclusive to a sequence with Protoss Robotics Facility only). This sequence 
unlocks detector technology (Observers are detectors, produced out of the Robotics Facility), 
so its probability variable (RO) should be conditioned on the belief that the opponent has 
invisible units (P(RO = true\Opening = DarkTemplar) = high or parametrize P(RO\ETT) 
adequately). The planner would both use resources planning, time of construction, and this 
posterior probabilities on sequences. 
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9.2.4 Inter-game Adaptation (Meta-game) 

In a given match, and/or against a given player, players tend to learn from their immediate 
mistakes, and they adapt their strategies to each other's. As this can be seen as a continuous 
learning problem (reinforcement learning, exploration-exploitation trade-off), there is more to 
it. Human players call this the meta-game*, as they enter the "I think that he thinks that I 
think..." game until arriving at fixed points. This "meta-game" is closely related to the balance 
of the game, and the fact that there are several equilibriums makes the interest of StarCraft. 
Also, clearly, for human players there is psychology involved. 



Continuous learning 



For all strategic models, the possible improvements (subsections 7.5.3 7.6.3, 7.7.2) would include 
to learn specific sets of parameters against the opponent's strategies. The problem here is that 
(contrary to battles/tactics) there are not much observations (games against a given opponent) to 
learn from. For instance, a naive approach would be to learn a Laplace's law of succession directly 
on P(ETechTrees = ett\Player = p) = #ETT+™bgamlf(p) ' anc ^ ^° * ne same for EClusters, 
but this could require several games. Even if we see more than one tech tree per game for the 
opponent, a few games will still only show a sparse subset of ETT. Another part of this problem 
could arise if we want to learn really in- game as we would only have partial observations. 



Manish Meta 2010 approached "meta-level behavior adaptation in RTS games" as a mean 



for their case-based planning AI to learn from its own failures in Wargus. The opponent is not 
considered at all, but this could be an interesting entry point to discover bugs or flaws in the 
bot's parameters. 



Bot's psychological warfare 

Our main idea for real meta-game playing by our AI would be to use our models recursively. As 
for some of our models (tactics & strategy), that can be used both for prediction and decision- 
making, we could have a full model of the enemy by maintaining the state of a model from them, 
with their inputs (and continually learn some of the parameters). For instance, if we have our 
army adaptation model for ourselves and for the opponent, we need to incorporate the output 
of their model as an input of our model, in the part which predicts the opponent's future army 
composition. If we cycle (iterate) the reasoning ("I will produce this army because they will have 
this one"...), we should reach these meta-game equilibriums. 



Final words 

Finally, bots are as far from having a psychological model of their opponent as from beating the 
best human players. I believe that this is our adaptability, our continuous learning, which allows 
human players (even simply "good" ones like me) to beat RTS games bots consistently. When 
robotic AI will start winning against human players, we may want to hinder them to have only 
partial vision of the world (as humans do through a screen) and a limited number of concurrent 



actions (humans use a keyboard and a mouse so they have limited APM*). At this point, they 
will need an attention model and some form of hierarchical action selection. Before that, all the 
problems arose in this thesis should be solved, at least at the scale of the RTS domain. 
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Glossary 



AI directors system that overlooks the behavior of the players to manage the intensity, diffi- 



culty and fun. 17 



APM action per minute, an input speed frequency in competitive gaming. 38 101 178 



Bayesian game a game in which information about knowledge about payoffs is incomplete. 
[28) [34 



BIC Bayesian information criterion, a score for model selection. For a given model with n data 



points, k parameters and L the maximum likelihood, BIC = — 21n(L) + fcln(n). 126 128 
E56] 

BP Bayesian program. [47] 

branching factor (average) number of nodes at each level of a search tree, i.e. base b of the 

complexity of a search of depth d in a tree, which is o(b d ). [Ti) |i}g g 

build order a formal specification of timings (most often indexed on total population count) 
at which to perform build actions in the early game.. [61] |122| |125| 

build tree abbrev. for "buildings tree", state of the buildings (and thus production) unlocked 
by a player. [59] [119] [mHm] [HTT] [T39j [T43] [147] [148] [161] [T74] 



BWAPI Brood War Application Programmable Interface. 101 



BWTA BroodWar Terrain Analyser. 97 205 



CBR Case-Based Reasoning. 96 122 176 



Dark Spore a fast-paced, sci-fi action-RPG, with PvP and cooperative (vs AI) modes. 17 



DSL Domain Specific Language. 18 



EM expectation-maximization, an iterative method to optimize parameters of statistical mod- 
els depending on unobserved variables. The expectation (E) step gives the likelihood 
depending on the latent variables, and the maximization (M) step computes maximizing 
parameters for the expectation of this likelihood.. 125 127[ |163| 



expand either placement of a new base or the action to take a new base (to collect more 



resources).. 67 167 175 
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fog of war hiding of some of the features (most often units and new buildings) for places where 
a player does not have units in range of sight.. 16 33 38, 60, 61 66 98] |203 



FPS First Person Shooter: egocentric shooter game, strong sub-genres are fast FPS, also called 
"Quake-likes", e.g. Quake III; and team/tactical FPS, e.g. Counter-Strike, Team Fortress 



2. fT6l [321 [371 [381 |40l |44l [9T 



FSM finite state machine. 18 30 79, 121 160, 161 



gameplay describes the interactive aspects of game design, which constraints the players possi- 
ble behaviors and the players' goals in a given game. Also, the category/type of the game.. 



15 16 28 59 60, 91 



gamification the use of game mechanics and game design to enhance non-game contexts. 17 



gas also called vespene, short for vespene gas. Advanced resource in StarCraft, use for all 
advanced units, technological buildings, upgrades and researches. It can be harvested only 
once an assimilator /refinery /extractor has been built on the gas geyser and maximally fast 
with 3 workers only at the closest (standard) distance to the resource depot.. [59] [6TJ [62] 

ED2 

GMM Gaussian mixture model, a generative model in which observations are the outcomes of 



mixtures of normal distribution.. 125 126 174 



goban board used for the game of Go. 23 



HFSM hierarchical finite state machine. 18 81 



HMM Hidden Markov Model, a dynamic Bayesian network estimating hidden states following 



a Markov process from observations. 44 122 



HTN hierarchical task network. 18,30 122 



Left 4 Dead a teamplay (cooperative vs AI) survival horror FPS in which players have to fight 
and escape zombie hordes. [17] 



match-up the pairing of two factions in StarCraft for a match in the form XvY, with X and 
Y being factions. When we consider only one race's strategy, it should be in front. For 
instance, PvT is the Protoss versus Terran match-up, with Protoss being studied if the 



features are not symmetric. 124,126 129 135 142 146,147 154 



max supply also total supply, maximum number of units that a player can control at a given 



time, can be increased up to a hard limit (200 in StarCraft).. 61 62 102 167 
MCMC Monte-Carlo Markov chain. [115] 



MCTS Monte-Carlo Tree Search. 23 24 115 



MDP Markov decision process. 37 38 
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meta-game preparation and maneuvering before (and between) games to exploit: current 
trends, opponent's style and weaknesses, map's specificities (safe zones, back doors, tim- 



ings), and psychological "mind games".. 178 



micro-management gameplay actions that are manually addressed by the player; the way of 
maximizing one's army efficiency by using the units to the maximum of their potential.. 
59j [65] [67} [7TH731 [78} [209] 



mineral basic resource in StarCraft, used for everything. It can be harvested increasingly faster 



with the number of workers (up to an asymptote) . . K39| Wu W2A |102 



mini- map radar view of the full game area, shown in the bottom corner of the interface in 
StarCraft.. [62] 

MMORPG Massively Multi-player Online Role Playing Game, distinct of RPG by the scale 
of cooperation sometimes needed to achieve a common goal, e.g. Dark Age of Camelot, 



World of War craft. 16,43 48 49 



NPC non-playing characters: game AI controlled third party characters, which were not con- 



ceived to be played by humans as opposed to "bots". 15-17 57 



opening in Chess as in RTS games: the first strategic moves of the game, the strategy of the 



61 


63 


66 


121 


123 


125 


142 


161 


162 
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partisan (game) which is not impartial, in which a player can do actions another can not do 
(move a faction while the other player(s) cannot). |2l] 

perfect-information (game) in which all the players have complete knowledge of the (board) 
state of the game. [21] 



POMDP partially observable Markov decision process. 38 71 , 94 



positional hashing a method for determining similarities in (board) positions using hash func- 
tions. El 



pro-gamer professional gamer, full-time job. 16 60 



PvE Players vs Environment. 17 31 49 



PvP Players versus Players. 17 



replay the record of all players' actions during a game, allowing the game engine to recreate 



the game state deterministically. [16 60, 101 124 



replayability replay value, entertainment value of playing the game more than once. 17 



RL reinforcement learning. 74 



RPG Role Playing Game, e.g. Dungeons & Dragons based Baldur's Gate. 16 32 38, 40 91 
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RTS Real-Time Strategy games are (mainly) allocentric economic and military simulations from 
an operational tactical/strategist commander viewpoint, e.g. Command & Conquer, Age 



of Empires, StarCraft, Total Annihilation. 11 12 16,37 38 40 



rush quick aggressive opening. 123 



solved game a game whose outcome can be correctly predicted from any position when each 



side plays optimally. 19 



StarCraft: Brood War a science fiction real-time strategy (RTS) game released in 1998 by 
Blizzard Entertainment. [16] 

supply cost in population (or supply) of a given unit, or current population count of a player. 



Originally, the Terran name for pop ulation/psi/control.. 61,62 102 167 



tech tree abbreviation for "technological tree", state of the technology (buildings, researches, 



upgrades) which are unlocked/available to a given player.. 59, 61 63, 68 98-100 113 
[TTUHT^l PHI [TdT] [Wl [TTT] [TT41 1207 



UCT Upper Confidence Bounds for Trees. 24 115 



zero-sum game a game in which the total score of each players, from one player's point-of- 
view, for every possible strategies, adds up to zero; i.e. "a player benefits only at the 
expense of others". [19] 
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Appendix A 

Game AI 



A.l Negamax 



Algorithm 7 Negamax algorithm (with closures) 
function negamax (depth) 
if depth < 0 then 
return value{) 
end if 
a < oo 

for all possible moves do 

a <— max(a, —negamax (depth — 1)) 
end for 
return a 
end function 



A. 2 "Gamers' survey" in section 

Questions 

How good are you? 

• Very good 

• Good 

How important is the virtuosity? (to win the game) reflexes, accuracy, speed, "mechanics" 

• 0 (not at all, or irrelevant) 

• 1 (counts, can be game changing for people on equal level at other answers) 

• 2 (counts a lot, can make a player win even if he is a little worse on lower importance 
gameplay features) 

How important is deductive thinking? (to win the game) "If I do A he can do B but not C" 
or "I see E so he has done D", also called analysis, forward inference." 
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• 0 (not at all, or irrelevant) 

• 1 (counts, can be game changing for people on equal level at other answers) 

• 2 (counts a lot, can make a player win even if he is a little worse on lower importance 
gameplay features) 

How important is inductive thinking? (to win the game) "He does A so he may be thinking 
B" or "I see D and E so (in general) he should be going for F (learned) ", also called generalization, 
"abstraction". 

• 0 (not at all, or irrelevant) 

• 1 (counts, can be game changing for people on equal level at other answers) 

• 2 (counts a lot, can make a player win even if he is a little worse on lower importance 
gameplay features) 

How hard is decision-making? (to win the game) "I have options A, B and C, with regard 
to everything I know about this game, I will play B (to win)", selection of a course of actions. 

• 0 (not at all, or irrelevant) 

• 1 (counts, can be game changing for people on equal level at other answers) 

• 2 (counts a lot, can make a player win even if he is a little worse on lower importance 
gameplay features) 

You can predict the next move of your opponent: (is knowledge of the game or of the 
opponent more important) 

• 1 by knowing what the best moves are / the best play for him? 

• -1 by knowing him personally (psychology)? 

• 0 both equal 

What is more important: 

• 1 knowledge of the game (general strategies, tactics, timings) 

• -1 knowledge of the map (specific strategies, tactics, timings) 

• 0 both equal 

Results 
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Appendix B 

StarCraft AI 

B.l Micro-management 



Algorithm 8 (Simplified) Target selection heuristic, efficiently implemented with a enemy units 
■H- damages bidirectional map: bimap : u,d — > (left : u — > d, right : d — > u) 

function ON _ DEATH (itnit) 

remove _incoming _damages(unit. tar get ^ damages(unit, unit.target)) 

end function 

function REGlSTER_TARGET(urat, target) 

add _incoming _damages(target, damages(unit, target)) 

unit.target target 
end function 

function select _TARGET(itnit) 

for all eunit G f ocus _ fire _order (enemy _units) do 
if eunit.type £ priority _targets(unit.type) then 
if in_range(unit, eunit) then 

register _target(unit, eunit) 
else if unit. prio _tar get == NULL then 

unit. prio_tar get eunit 
end if 
end if 
end for 

if unit.target == NULL then 

for all eunit G f ocus _ fire _order (enemy _units) do 
if in_range(unit, eunit) then 

register _target(unit, eunit) 
end if 
end for 
end if 

if unit.target == NULL and unit. prio_tar get == NULL then unit. pr io _tar get 
closer(unit, enemy _units) 

end if 
end function 
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Figure B.l: Plate diagram of a Bayesian unit with ./V possible directions. 



B.2 Tactics 



frame 


player name 


player id 


order / action 


X pos. 


Y pos. 


unit 


6650 


L.Nazgul[pG] 


0 


Train 






Probe 


6720 


SaFT.eSu 


1 


Train 






Probe 


6830 


L.Nazgul[pG] 


0 


Train 






Probe 


7000 


SaFT.eSu 


1 


Train 






Probe 


7150 


L.Nazgul[pG] 


0 


Build 


39 


108 


Forge 


7245 


L.Nazgul[pG] 


0 


Build 


36 


108 


Citadel of Adun 


7340 


L.Nazgul[pG] 


0 


Train 






Probe 


7405 


SaFT.eSu 


1 


Train 






Probe 


7415 


L.Nazgul[pG] 


0 


Train 






Probe 


7480 


SaFT.eSu 


1 


Train 






Shuttle 


7510 


SaFT.eSu 


1 


Build 


26 


24 


Robotics Support Bay 



Table B.l: Example of what is in a replay (in human readable format), lines with non-empty 
positions are constructions of buildings. Several other action types are not represented (current 
selection, move/attack orders...) 



B.2.1 Decision-making: soft evidences and coherence variables 

Here we will present the full tactical model for decision making, with soft evidences of variables 
we know only partially (variables we have only the distribution). 

Variables 



In the tactical model (section 6.5), for some variables, we take uncertainty into account with 
"soft evidences": for instance for a region in which no player has a base, we have a soft evidence 
that it belongs more probably to the player established closer. In this case, for a given region, 
we introduce the soft evidence variable(s) B' and the coherence variable A# and impose P(A^ = 



1\B,B') = 1.0 iff B = B', else P(A B = l\B,B') = 0.0; while P{X B \B, B')P(B') is a new factor 
in the joint distribution. This allows to sum over P(B') distribution (soft evidence). We do that 
for all the variables which will not be directly observed in decision-making. 



Decomposition 

The joint distribution of our model contains soft evidence variables for all input family variables 
(E, T, B, GD, AD, ID) as we cannot know for sure the economical values of the opponent's 
regions under the fog of war} " (E), nor can we know exactly the tactical value (T) for them, 



nor the possession of the regions (B), nor the exact defensive scores (GD, AD, ID). Under this 
form, it deals with all possible uncertainty (from incomplete information) that may come up in 
a game. For the n considered regions, we have: 

P(Ai- n , E\ vn , T\-. n , T A\-, n , B\ M , B[. n , Xb,1:ti} T[. n , Ar,l:n, (B-1) 

El: n , A_B,l:n, ID' Vn , A/D,l:n, GD[. n , XcD,l:n, AD' 1:n , \AD,l:n, (B-2) 

H 1:n , GD lm , AD 1:n , ID 1:n , HP, TT) (B.3) 

n 

= HlPiAi^iEi^TA^BilAi) (B.4) 

i=i 

P(X Bji \B lm ,B[jP(B[.jP(X T>i \T lm , T{.JP(T[ :n ) (B.5) 

P(X E ,i\E 1:n , E' 1:n )P(E' Un )P{X ID!i \ID 1:n , ID[, n )P(ID' 1:n ) (B.6) 

P(X GDti \GD lm , GD'^JPiGDlJPiXADMDi.n, AD[. n )P(AD[. n ) (B.7) 

P(AD h GDi, IDi\Hi)P(Hi\HP)] P(HP\TT)P(TT) (B.8) 



The full plate diagram of this model is shown in Figure B.6 



Forms 



To the previous forms (section 6.5.1 ), we add for all variables which were doubles (X with X'): 



P(X X \X,X') = 1.0 iff X = X' 
P{X X \X,X') = 0.0 else 



Identification 



The identification and learning does not change, c.f. section 6.5.1 

Questions 

Vi G regions P(Ai\tai, Xbj = 1, Xtj = 1, ^E,i = 1) 

P(Ek, Ti,tai, fl 4 |Ai)P(i4i)P(B0P(#)P(£S) 



Bi,B[ JT i ,T' i J E i ,E' i 

Vi G regions Y > {H i \tt,\i D ^ = l,\oD,i = l,^AD,i = 1) 

y2P(AD l ,GD l ,ID l \H l )P{H l \HP)P(HP\tt)P(ID' l ,)P{AD' 1 )P{GD' l ) 

IDiJD'. JGD^GD'. J AD i, AD'. Hp 
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Bayesian program 

The Bayesian program of the model is as follows: 

Variables 

Al; n , E\- n , T\- n , T A\- n , Bl.n, B[. n , A_B,l:n, T[. n , \t,1:u, E' 1:n , Xe,1:ti, 

ID[. n , \lD,l:n, GD[. n , \GD,l:n, AD' 1:n , \AD,l:n, #1:71, G D\- n , AD\- n , 1 D\- n , HP, TT 

Decomposition 

P(Al :n , El-.n, Ti :n , T A\- n , Bl;„, B' 1]n , A_B,l:n, T[. n , \t,1:ti, E' 1:n , \E,l:n, 

ID' ln , XlD,l:n, GD[. n , \GD,l:n, AD' 1:n , \AD,l:n, Hl :n , GD\ :n , AD\ :n , ID\. n , HP, TT) 

= nr=i [P(A 1 )P(E,,T,TA l ,B l \A l ) P(A s , i |Si : „,^ : „)P( J B; :n )P(A T , i |T 1:TO; T 1 ' : „)P(n n ) 
P(A B , i |£i ! „,tf l!n )P(Ei ! „)P(Aj D , i |/iJi:„,^ 

P(\ADMADi: n ,AD' 1:n )P(AD' 1:n ) P(ADi,GDi,IDi\Hi)P{Hi\HP)]P(HP\TT)P{TT) 
Forms 

P(A r ) prior on attack in region i 
P(E,T, TA,B\A) covariance/probability table 
P(Xx\X,X') = 1.0 iff X = X', else P(X X \X,X') = 0.0 (Dirac) 
P(AD,GD, ID\H) covariance/probability table 
P(H\HP) = Categorical(4,HP) 

P(HP = hp\TT) = 1.0 iff TT -> ftp, e/se P(HP\TT) = 0.0 
_P(TT) comes from a strategic model 
Identification (using 5) 



O 

Sh 

a 

o 

pq 



■r S 



S < 

o 



P(A r = true) = 



^battles/ ga 



(probability to attack a region) 



n battles+ n not battles ^regions/ 

it could be learned online (preference of the opponent) : 
P(A r — true) = 2+J2 1+ " ba "'"^ ; — j^y (online for each game) 
P(E = e,T = t,TA = ta,B = b\A = True) - i+n battll!S (e,t,ta,b) 

P{AD = ad, GD = gd, ID = id\H = h) = lADlxlG D Mm+ ^ GDIDnbattlcA A D ,GD, ID , h) 
P(H = h\HP = hp) - — l+mMUeA!±M 



|S|x|T|x|TA|x|S|+X; EiTiTj 4 iB n battle3 (E,T,TA,B) 
^- J r ri battles (o,d,gd,id,h) 



\H\+J2 H n battles (H,hp) 

Questions 
decision — making 

Vi G regions? (Ai\tai, As,, = 1, Xr,i = 1, Xe,i = 1) 
yi G regionsP(Hi\tt, X ID ,i = 1,Xgd,i = 1,Xad,{ = 1) 
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Figure B.2: Top: StarCraft's Lost Temple map (one of the most famous maps with Python). 
We can see features like cliffs, ramps, walls, waters and resources (minerals and gas). Bottom: 
output of BWTA" with the regions slicing. We can see regions with one or several chokes, but 
also isolated regions as gray is non walkable terrain (crossable by flying units only). 
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[Replay Start] 
RepPath: $ (PATH/TO/REP) 
MapName: $MAPNAME 
NumStartPositions : $N 

The following players are in this replay: 
<list of 

$PLAYER_ID, $PLAYER_NAME , $START_L0C 
separated by newlines> 
Begin replay data: 
<list of 

$FRAME_NUMBER , $PLAYER_ID , $ACTI0N , [$ACTION_DEP_ARGS] 

separated by newlines> 

[EndGame] 



Figure B.3: Template of an RGD (replay general data) file from the dataset (see section 6.4) 



<list of 

$FRAME, $UNIT_ID , $0RDER, Target OrPos it ion , $P0S_X , $P0S_Y 
separated by newlines> 



Figure B.4: Template of an ROD (replay order data) file from the dataset (see section 6.4) 



Regions , $REGIONS_IDS_COMMA_SEPARATED 
$REGI0N_ID, $DIST, $DIST, ... 
$REGI0N_ID, $DIST, $DIST, ... 



ChokeDepReg, $REGIONS_IDS_COMMA_SEPARATED 
$REGI0N_ID, $DIST, $DIST, ... 
$REGI0N_ID, $DIST, $DIST, ... 



[Replay Start] 
<list of 

$FRAME, $UNIT_ID, $P0S_X, $P0S_Y 
$FRAME , $UNIT_ID , Reg , $REGI0N_ID 
$FRAME, $UNIT_ID , CDR , $CDR_ID 
separated by newlines> 



Figure B.5: Template of an RLD (replay location data) file from the dataset (see section 6.4) 
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Figure B.6: Plate diagram (factor graph notation) of the Bayesian tactical model in decision- 
making mode. Hatched nodes are nodes on which we only have a distribution. We know T A 



(our tactical scores as we are the attacker) and TT (our tech tree"). 
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B.3 Strategy 

B.4 BroodwarBotQ 



Algorithm 9 Flow algorithm making sure there are no convex closures 
function WYv(region) 

{dirij <— list()\(i,j) £ region} 
updated <— {(i, j)\(i, j) £ entrance(region)} 
{dirij <— dir _towards(region)\(i, j) £ updated} 
while Bdivij == listQ do 

(sources, new _updated) <— neighbours _couples(updated) 
for all ((x, y), (i, j)) £ (sources, new _updated) do 
if (x — i,y — j) ^ dir X) y then 

dirij .append((i — x, j — y)) 
end if 
end for 

updated <— new _updated 

end while 
end function 
function BUILD (i,j) 

refill listQ 

for all (x,y) £ neighbour s(i,j) do > cut the flow arround 

dir x ^ y .remove((x — i,y — j)) 
if dir X: y. empty () then 
refill.append((x, y)) 

build(x, y) > Recursively cut the flow 

end if 
end for 

while -i j 'ixed _point do > refill as in the initialization... 

current <— dir 
for all (x, y) £ refill do 

if 3(i, j) £ neighbour s(x,y) such that currently / listQ then > non empty flow 

dir XtV .append((x - i,y - j)) 
end if 
end for 
end while 
end function 

function is_isolated?(z, j) 
if dirij. empty () then 

return True 
else 

return False 
end if 
end function 
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Filling: 1, 2, 3, 4. 



Empyting: 1, 2, 3, 4 



Figure B.7: Example of the algorithm [9] in a case in which there are buildings A, B, C, D and 
building in the new cross spots would close the paths to the interior and trap units. Left: flood 
filling of the region with the source at the choke (bottom right). The arrows show the dir vectors 
for each tile. Right: recursive calls to build which drain the interior zone. Refilling will not be 
able to refill the middle. 



# 


Ik.l 


Race 


ELO rating 


Total games 




Draws 


Crashes 


Wins w/o 
crashed games 


Losses w/o 
crashed games 


BWAPIrev 


Description 


Status 


I 


Skynel_v2J)J 




2156 


1493 


1300(87.07%) 


189 




0(0%) 


1175 (85.89%) 


189 


4025 




enabled 


2 


UAIlKrtaBot_bwapi_402S 




21 30 


153 


135 (88.24%) 


17 




1 (0.65%) 


129 (88.36%) 


16 


4025 




enabled 


~3 


UAlbertaBot_aiide_201 1 




2105 


1890 


1434 (75.87%) 


404 




109 (5.77%) 


1402(80.16%) 


295 


3769 




disabledNewerVersicn 


1 


Skynet_aiide_2011 


* 


2005 
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690(8175%) 
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20 (2.37%) 


603 (81.82%) 
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3769 




disabledNewerVersicn 
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Aiur_bwapi_4000 


# 


1944 


1389 


939 (67.6%) 


409 




9(0.65%) 


886 (66.77%) 


400 


4025 




enabled 


~6 


Aiur_aiidc_201 1 


& 


1931 


1912 


1337 (69.93%) 
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11 (0.58%) 


1242(68.77%) 
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3769 




enabled 


1 


ItayUndennind_aiidc_2<>ll 


1 


1895 


1742 


1056 (60.62%) 
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0(0%) 


968 (58.52%) 
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3769 




enabled 


~S 


krasi0_2_15 


-ft 


1881 
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0(0%) 


502 (63.79%) 
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4025 




enabled 


~9 


MaehineJLll 


* 


1863 
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36 (3.64%) 
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270 


4025 




enabled 


To 


krasi0_2_14 


H 


1841 
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214 




1 (0.12%) 


483 (66.71%) 


213 


4025 




dlsabledNewer Version 


IT 


Overmind_audc_2010 
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1796 
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872 (67.34%) 
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839 (66.533) 


385 


2423 




enabled 


12 


BroodwarBotQ_bwapi_4WW 


» 
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Figure B.8: Bots ladder on February 12th, 2012. With BroodwarBotQ using a Bayesian 



model for opponent's strategy prediction as well as for micro-management 
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Nani gigantum humeris insidentes. 
Des nains sur des epaules de geants. 
Standing on the shoulders of giants. 



